Digital light processing hyperspectral imaging apparatus

ABSTRACT

A hyperspectral imaging system having an optical path. The system including an illumination source adapted to output a light beam, the light beam illuminating a target, a dispersing element arranged in the optical path and adapted to separate the light beam into a plurality of wavelengths, a digital micromirror array adapted to tune the plurality of wavelengths into a spectrum, an optical device having a detector and adapted to collect the spectrum reflected from the target and arranged in the optical path and a processor operatively connected to and adapted to control at least one of: the illumination source; the dispersing element; the digital micromirror array; the optical device; and, the detector, the processor further adapted to output a hyperspectral image of the target. The dispersing element is arranged between the illumination source and the digital micromirror array, the digital micromirror array is arranged to transmit the spectrum to the target and the optical device is arranged in the optical path after the target.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims priority to U.S. Provisional PatentApplication No. 61/087,714, filed Aug. 10, 2008, U.S. Provisional PatentApplication No. 61/087,715, filed Aug. 10, 2008, and U.S. ProvisionalPatent Application No. 61/168,347, filed Apr. 10, 2009, whichapplications are incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Grant Number 5 U24DK076169-02 awarded by the National Institutes of Health. The governmenthas certain rights in the invention.

FIELD OF THE INVENTION

The present invention relates generally to the field of spectroscopicimaging that results in collecting/measuring spectroscopic image dataand the associated algorithms that result in chemically encoded imagesof tissue, including devices, methods and systems for fluorescenceimaging of tissues in real-time. Additionally, the present inventionrelates generally to the field of hyperspectral imaging of medicaltissues. In particular, the invention is directed to new devices, toolsand processes for the in vivo detection and evaluation of diseases anddisorders. Moreover, the present invention describes, but is not limitedto, a spectrally conformable illumination system and associatedanalytical algorithms, e.g., chemometrics, spectral and image digitalsignal processing, used to reduce the number of image frames required togenerate a real-time processed image that is color-coded based onmatching the reflectance of each pixel in an image to known reflectancespectra. Generally, the present invention is applicable to surgical andclinical applications, and is also applicable to a variety of otherapplications, including but not limited to microscopy, pathology, foodsafety/inspection and pharmacology.

BACKGROUND OF THE INVENTION

Without limiting the scope of the invention, its background is describedin connection with laparoscopic procedures, such as cholecystectomy, inconnection with a hyperspectral imaging system used for medical tissuesand in connection with new devices, tools and processes for the in vivodetection and evaluation of diseases and disorders.

The gallbladder is a small pear-shaped organ that stores andconcentrates bile. The gallbladder is connected to the liver by thehepatic duct. It is approximately 3 to 4 inches (7.6 to 10.2 cm) longand about 1 inch (2.5 cm) wide. The function of the gallbladder is tostore bile and concentrate. The bile emulsifies fats and neutralizesacids in partly digested food. A muscular valve in the common bile ductopens, and the bile flows from the gallbladder into the cystic duct,along the common bile duct, and into the duodenum (part of the smallintestine).

The following are disorders of the gall bladder. Cholelithiasis is adisorder of the extrahepatic biliary tract related to gallstonesresulting in the presence of stones in the gall bladder. Gall stones arebodies formed within the body by accretion or concretion of normal orabnormal bile components. Gall stones of various shapes and sizes areformed within the gall bladder. Cholecsytitis is the inflammation of thegall bladder. Cholecystitis is often caused by Cholelithiasis, withcholeliths most commonly blocking the cystic duct directly which causesthe gallbladder's wall to become inflamed. Extreme cases may result innecrosis and rupture, and could cause further infection and painresulting from such inflammation. Inflammation often spreads to itsouter covering, irritating surrounding structures such as the diaphragmand bowel. Cholecystitis usually presents as a pain in the right upperquadrant. Gall bladder Cancer is a relatively uncommon cancer thatoccurs in the gall bladder. If detected early, this cancer can be curedby removing the gall bladder.

Gall bladder diseases are marked with some or all of the followingsymptoms: severe and constant pain in the upper right abdomen which canlast for days; increasing pain when drawing a breath; and, radiatingpain to the back or occurring under the shoulder blades. About a thirdof patients have fevers and chills. Nausea and vomiting may also occur.Complaints of gas, nausea, and abdominal discomfort after meals are themost common, but they may be vague and indistinguishable from similarcomplaints in people without gallbladder disease. Moreover, gall bladderdiseases may result in jaundice (yellowish skin), dark urine, lighterstools, or combinations thereof.

As the majority of patients incur gallstones along with a gall bladderdisease, the diagnosis can usually be confirmed thorough ultrasoundimaging, a safe, painless and non-invasive technique that uses highfrequency sound waves to create an image of gallbladder and gallstones.Other diagnosis methods like X-Ray and other scanning technology may beused.

Cholecystectomy is one of the most common operations performed in UnitedStates. It is frequently used for the treatment of symptomaticgallstones. Cholecystectomy is the surgical removal of gall bladder. Thetwo procedures utilized to surgically remove the gall bladder are OpenCholecystectomy and Laparoscopic Cholecsytectomy. The laparoscopicmethod is utilized more frequently. The choice of the procedure is madeon individual basis. A Cholecystectomy is performed to treatCholelithiasis and Cholecystitis.

In conventional Cholecsytectomy, a surgeon makes an incisionapproximately 6 inches long. The incision is made either longitudinallyin the upper portion of the abdomen or obliquely beneath the ribs on theright side. During the procedure, drains may be inserted into theabdomen, which are usually removed while the patient is still in thehospital. Following a normal cholecystectomy, the patient may be in thehospital from one to three days post surgery. Normal activity can beresumed in about four weeks. In complicated cases normal activity can beresumed in about four to eight weeks. The procedure is very common andis successful most of the time. However, any surgery involves riskfactor associated with it, which can cause complications. The mostcommon associated risk with Cholecsytectomy is the injury to the commonbile duct (CBD) which is hidden, as it lies below a layer of fat. Thus,it is the surgeon's expertise and judgment for locating the CBD andavoiding any injuries.

A laparoscope is a long and rigid tube that is attached to a camera anda light source. Before the laparoscope is inserted, the patient'sabdomen is distended with an injection of carbon dioxide gas, whichallows the surgeon to see the internal organs of the patient. With thehelp of the laparoscope and a video display, which guides the surgeonfor the procedure, the surgeon is able to locate and perform acholecystectomy. Other small incisions are made in the abdomen; two ofthem on the right side below the rib cage and one in the upper portionbelow the sternum or the breast. Many other sophisticated instrumentsare used to perform the procedure. For example, two instruments are usedto grasp and retract the gall bladder and a third is used to free thegall bladder from its attachment. Once the gall bladder is free, thesurgeon then removes it from the patient.

Under normal conditions, patients recover within a day or two; however,complications may still occur. The most common complication that canoccur is the injury to common bile duct (CBD). This complication occursin about 0.5% of cholecystectomies. Even though this complication israre, it demands a better imaging technique that provides the surgeonwith the precise location of the CBD during a cholecystectomy. Theimaging technique should also be able to distinguish between the CBD andother tissue, providing a good contrast image.

During a cholecystectomy, one of the important problems faced by thesurgeon is visibility of structures below the fat layer. Thus, it is thesurgeon's experience which determines the approximate location of thebile duct. An imaging aid currently available for intraoperative bileduct visualization is an Intraoperative Cholangiography (IOC). Thistechnique has hardly undergone substantial changes since itsintroduction by Mirizzi in 1937. (See Mirizzi P L. 1937 OperativeCholangiography Surg Gynaecol Obstet. 1937; 65; 702-710). Routine IOCperformed with every cholecystectomy is one of the strategies used toreduce bile duct injuries. The rapidly advancing technology of nuclearimaging, diagnostic ultrasound, MRI and CT have also been used for thepurpose of visualizing the bile duct. For diagnosis of hepatobiliarydisease imaging techniques such as ultrasound and Magnetic ResonanceCholangiography (MRC) are frequently used. Endoscopic RetrogradeCholangiography (ERC) is another standard for visualization of the bileduct. Ultrasound is tolerated by patients and is cost effective. MRC issuperior in visualizing the biliary system, does not require anycontrast agent to visualize the bile ducts and dilatation and gallstonesin CBD are easily detected. For patients with choedocholithiasis,endoscopic retrograde cholangio-pancreatography (ERCP) can be utilized.Selective use of preoperative ERCP has proven to be a good diagnostictool, as well as a way to allow clearance of CBD stones when present.(See Qasim Al-qasabi et al. “Operative Cholangiography in LaproscopicCholecystectomy: Is it essential”). One of the faster and widelyavailable techniques is Computed Tomography Cholangiography (CTC). Amulti-detector CT reports a sensitivity of 65%-88% and a specificity of84%-97% to detect gallstones. With the development of multi-detetcor CT,the resolution of CTC exceeds that of MR. (See A Persson, N Dahlstrom, ÖSmedby and T B Brismar: “Three-dimensional drip infusion CTcholangiography in patients with suspected obstructive biliary disease:a retrospective analysis of feasibility and adverse reaction to contrastmaterial”).

Most histological evaluation of living tissues typically involvesfixation, sectioning, and/or staining to obtain samples which exhibithigh quality images under microscopy. While this process is current theindustry standard, the procedure is time consuming. It requires removalof tissue from the patient, processing time, and has inherent samplingerror. In addition, the major limitation of this process is the delay inproviding the surgeon with clinically relevant information at the timeof surgery. Thus, new methods have been developed to complement existingmodalities by providing the surgeon real-time information that could beused intra-operatively to identify suspect lesions.

One such method is hyperspectral imaging. Hyperspectral imaging is amethod of imaging spectroscopy that generates a gradient map based onlocal chemical composition. Hyperspectral imaging has been used insatellite investigation of suspected chemical weapons production areas,geological features, and the condition of agricultural fields, and hasrecently been applied to the investigation of physiologic and pathologicchanges in living tissue in animal and human studies. Hyperspectralimaging has also been used in medical applications and has been shown toaccurately predict viability and survival of tissue deprived of adequateperfusion, and to differentiate diseased (e.g., tumor) and ischemictissue from normal tissue.

One such example can be seen in United States Patent ApplicationPublication No. 2007/0016079 (Freeman et al.). The '079 applicationdescribes methods and systems of hyperspectral and multispectral imagingof medical tissues. In particular, the '079 application is directed tonew devices, tools and processes for the detection and evaluation ofdiseases and disorders such as diabetes and peripheral vascular disease,that incorporate hyperspectral or multispectral imaging.

Another example can be found in United States Patent ApplicationPublication No. 2007/0002276 (Hirohara et al.). The '276 applicationdescribes spectral characteristics that reduce variation depending onthe frequency of received light intensity, and is gentle on a subject'seye. The invention of the '276 application eliminates displacementbetween positions of respective spectral images of the same area even ifa change in alignment occurs between the eye and apparatus during imagecapture. In the '276 application, an apparatus for measuring spectralfundus image includes: an illumination optical system having anillumination light source that emits a light beam in a specifiedwavelength range; a light receiving optical system for forming a fundusimage on the light receiving surface of a photographing section; aliquid crystal wavelength tunable filter capable of choosing awavelength of a transmitted light beam in a specified wavelength range;a spectral characteristic correction filter having a wavelengthcharacteristic for correcting the wavelength characteristic of theemitted light intensity of the illumination light source and thetransmission wavelength characteristic of the wavelength tunable filterso that the received light intensity on the light receiving surface iskept within the specified range; and, a data measuring section fortaking the spectral fundus image data from the light receiving surfacewhile changing the wavelength of the light beam passing through thewavelength tunable filter.

Yet another example is described in U.S. Pat. No. 7,199,876 (Mitchell)entitled “Compact Hyperspectral Imager”. Here, the '876 patent details ahyperspectral imager including: a first optical sub-system; at least oneslit element; a second optical sub-system; at least one reflectivedispersive element located at a center plane; and, at least onedetecting element located at substantially an image surface. Duringoperation, the first optical sub-system images, onto the slitelement(s), electromagnetic radiation originating at a source. Thesecond optical sub-system substantially collimates, at a center plane,electromagnetic radiation emanating from the slit element(s). The secondoptical sub-system also images, onto the image surface, theelectromagnetic radiation reflected from the reflective dispersiveelement(s). The detecting element(s) detect the dispersedelectromagnetic radiation reflected from the reflective dispersiveelement(s).

Still yet another example is seen in U.S. Pat. No. 7,167,279 (Otten)entitled “High Efficiency Spectral Imager”. The '279 patent describesoptical instruments having, inter alia, optics to process wavelengths ofelectromagnetic radiation to produce an interferogram. The instrumentsdescribed in the '279 patent include at least one optical path andoptical elements positioned along this path for splitting andrecombining the wavelengths which interfere with each other to produce aplurality of different fringes of different wavelengths. In one group,the optics include matched gratings which are positioned along theoptical path outside of the interferometer optics to produce first andsecond sets of spectrally dispersed beams. The interferometer opticsalso include a beam splitter and first and second mirrors. The gratingsmay be positioned in a variety of locations along the optical path. Inanother group, the optics include a beam splitter having a plurality ofsurfaces, wherein each of the surfaces is either 100% reflective, 100%transmissive or 50% reflective and 50% transmissive. In a third group,the optics include the beam splitter having a plurality of reflectiveand transmissive surfaces and matched gratings. The instruments can allinclude a detector for detecting the interferogram and means forprocessing the detected interferogram to produce spectral information.

U.S. Pat. No. 6,198,532 (Cabib) discloses a spectral bio-imaging methodfor enhancing pathologic, physiologic, metabolic and health relatedspectral signatures of an eye tissue. The method disclosed in the '532patent includes the steps of: (a) providing an optical device for eyeinspection being optically connected to a spectral imager; (b)illuminating the eye tissue with light via the iris, viewing the eyetissue through the optical device and spectral imager and obtaining aspectrum of light for each pixel of the eye tissue; and, (c) attributingeach of the pixels a color according to its spectral signature, therebyproviding an image enhancing the spectral signatures of the eye tissue.

Another example can be found in U.S. Pat. No. 6,992,775 (Soliz et al.).The '775 patent discloses an ophthalmic instrument for obtaining highresolution, wide field of area hyperspectral retinal images for varioussized eyes which includes a fundus retinal imager (which includes opticsfor illuminating and imaging the retina of the eye); apparatus forgenerating a real time image of the area being imaged and the locationof the hyperspectral region of interest; a high efficiency spatiallymodulated common path; a Fourier transform hyperspectral imager (a highresolution detector optically coupled to the hyperspectral and fundusimager optics); and, a computer (which is connected to the real timescene imager, the illumination source, and the high resolution camera)including an algorithm for recovery and calibration of the hyperspectralimages.

Finally, in U.S. Pat. No. 7,118,217 (Kardon el al.) describes an opticalimaging device of retinal function to detect changes in reflectance ofnear infrared light from the retina of human subjects in response tovisual activation of the retina by a pattern stimulus. The device of the'217 patent measures changes in reflectance corresponding in time to theonset and offset of the visual stimulus in the portion of the retinabeing stimulated. Any changes in reflectance can be measured byinterrogating the retina with a light source. The light source may bepresented to the retina via the cornea and pupil or through othertissues in and around the eye. Different wavelengths of interrogatinglight may be used to interrogate various layers of the retina.Additionally, various patterns and methods of stimulation have beendeveloped for use with the imaging device and methods.

The aforementioned hyperspectral imaging systems are slow for routineclinical practice. In addition, there are no methods for directlyimaging the in vivo level of the biomolecules in live humans or animalsduring clinical visits or during surgical (open, endoscopic orlaparoscopic) operations. Accordingly, there is a need for an improvedmicroscopy system and method that incorporates superior speed forhyperspectral/multimodal imaging while offering high spatial resolutionand optimized signal sensitivity for fast image acquisition. The presentinvention is directed to such a need.

BRIEF SUMMARY OF THE INVENTION

In one embodiment, the present invention is a hyperspectral imagingsystem that includes one or more optical tunable radiation sourcesconfigured to illuminate one or more fluorescent targets; one or morelight dispersing elements positioned in the optical path between the oneor more optical tunable radiation sources and one or more detectors; oneor more spatial light modulators, capable of tuning light from theoptical radiation source into at least one spectral band or the bandspectrum, placed in the optical path before the one or more fluorescenttargets; an optical microscope configured to collect the at least onespectral band or the band spectrum reflected from the one or morefluorescent targets positioned in the optical path between the opticaltunable radiation sources and the one or more detectors; and a processorconnected to the one or more optical tunable radiation sources, the oneor more detectors to process the at least one spectral band or a bandspectrum reflected from the fluorescent target into image data, or both.In one aspect, the fluorescent target comprises natural fluorescence ofa tissue or fluid. In another aspect, the one or more optical tunableradiation sources produce electromagnetic radiations having wavelengthsover the range of 250 nm-2,500 nm. In another aspect, the one or moreoptical tunable radiation sources is a supercontinuum laser. In oneaspect, the hyperspectral imaging system is a microscope.

The system may further comprise at least one deconvolution algorithmthat normalizes each of the at least one spectral band or the bandspectrum at each pixel before image processing. In one aspect, thespatial light modulator is a digital micromirror device. In one aspect,the at least one spectral band or the band spectrum are processed at avideo rate. One or more dispersing elements may be a grating, a prism, atunable filter, an electromechanical optical filter wheel, anacousto-optical tunable filter, a liquid-crystal tunable filter, adigital micromirror device, or any combination thereof. One or moredetectors may be selected from the group consisting of: a spectrometer,a two-dimensional array detector, a multi-array detector, an on-chipamplification CCD camera, a back-illuminated CCD, a liquid nitrogencooled CCD detector or a focal plane array, e.g., a CCD, VisGaAs, InGasor ferromagnetic. In one aspect, the processor comprises an image dataacquisition software that tunes the spatial light modulator, triggersthe one or more detectors for collection of a series of spectroscopicimages formatted as a hyperspectral image cube and processes the imagedata for visualization; and one or more converters to digitize imagedata, wherein the processor is connect to one or more displays. Inanother aspect, the processor tunes the one or more spatial lightmodulators, triggers the one or more detectors for collection of aseries of spectroscopic images formatted as a hyperspectral image cubeand processes the image data for visualization.

In another aspect, the fluorescence is natural fluorescence, afluorescent dye, a fluorescence resonance energy donor, a fluorescenceresonance energy acceptor, a fluorescence quencher or combinationsthereof. Non-limiting examples of one or more fluorescence molecules foruse with the present invention include indocyanine green,5-carboxyfluorescein (5-FAM), 6-carboxyfluorescein (6-FAM),fluorescein-5-isothiocyanate (FITC),2′7′-dimethoxy-4′5′-dichloro-6-carboxyfluorescein (JOE); rhodamine andrhodamine derivatives such as N,N,N′,N′-tetramethyl-6-carboxyrhodamine(TAMRA), 6-carboxyrhodamine (R6G), tetramethyl-indocarbocyanine (Cy3),tetramethyl-benzindocarbocyanine (Cy3.5), tetramethyl-indodicarbocyanine(Cy5), tetramethyl-indotricarbocyanine (Cy7), 6-carboxy-X-rhodamine(ROX); hexachloro fluorescein (HEX), tetrachloro fluorescein TET;R-Phycoerythrin, 4-(4′-dimethylaminophenylazo) benzoic acid (DABCYL),and 5-(2′-aminoethyl)aminonaphthalene-1-sulfonic acid (EDANS) orcombinations thereof.

In another aspect, the processor is defined further as: a computerhaving image data processing software capable of processing variousfluorescence and digital signals; a digital signal processor comprisingalgorithms for fluorescence analysis; and, an algorithm library forvisualizing chemistry on and/or within the imaged one or morefluorescent targets for detection, monitoring and diagnosis. In oneaspect, the one or more illumination optics comprises lenses,microscopes, dissection microscopes, surgical microscopes, slit lamps,endoscopes, laparoscopes, colonoscopes or any combination thereof. Inanother aspect, the processor is in communication with a databasecomprising a library spectrum of fluorescent targets. In yet anotheraspect, the one or more fluorescent targets comprises a tissue, in thebloodstream, in the lymphatic system, in bodily secretions, in biopsies,on the skin, by fiber optic transdermal spectroscopy on a blood vessel,on the skin or at the retina, by laparoscopy, by endoscopy, in thecentral nervous system or combinations thereof. Moreover, in still yetanother aspect, the one or more fluorescent targets comprises an organ.In another aspect, the system can be configured for real-time in vivoanalysis of the one or more fluorescent targets.

Another embodiment of the present invention is an apparatus and methodfor obtaining spectral image data from one or more fluorescent targetsin vivo comprising: generating a spectrum of electromagnetic radiation;separating the electromagnetic radiation into at least one spectral bandor a band spectrum using a spatial light modulator; illuminating the oneor more fluorescent targets with the at least one spectral band or theband spectrum; collecting the reflected at least one spectral band orthe band spectrum resulting from the one or more fluorescent targets;and, directing the collected at least one spectral band or the bandspectrum to one or more detectors that capture spectral image data. Themethod may further comprise the steps of: tuning the spatial lightmodulator; triggering the one or more detectors for collection;formatting and digitizing the spectral image data as a hyperspectralimage; deconvoluting each pixel of the spectral image data using asignal processing algorithm; and, displaying the spectral image on oneor more fluorescent targets. In one aspect, the one or more fluorescenttargets comprise tissue components that have natural or nativefluorescence.

Another embodiment of the present invention is a method forhyperspectral surgery by capturing image data of one or more fluorescenttargets generating a wide spectrum electromagnetic radiation comprising:separating the electromagnetic radiation into at least one spectral bandor a band spectrum using a spatial light modulator; illuminating the oneor more fluorescent targets with the at least one spectral band or theband spectrum; collecting the reflected at least one spectral band orthe band spectrum resulting from the one or more fluorescent targets;and, directing the collected at least one spectral band or the bandspectrum to one or more detector to form the image data, wherein thehyperspectral surgery is used in surgical procedures selected from acholecystectomy, amputation, burn, skin flap evaluation, visualizingareas of angiogenesis, probes that bind antigens and absorbsnear-infrared during pathological evaluations and in vivo, qualitycontrol of pharmaceuticals, monitoring vascular changes and drugdiscovery in response to pharmaceuticals, monitoring diabeticretinopathy, diseases such as cancer, diabetes, sickle cell, anemia,bilirubin, raynauds, ulcers, burns, skin flaps, surgery, gallbladder,brain, monitoring wound healing, and early detection of woundinfections.

Another embodiment of the present invention is a method forhyperspectral surgery of a human subject by capturing image data of oneor more fluorescent targets generating a wide spectrum electromagneticradiation comprising: covering the human subject with NIR transparentmaterial to provide privacy, accessibility and maintain body heat;separating the electromagnetic radiation into at least one spectral bandor the band spectrum using a spatial light modulator; illuminating thefluorescent target with the at least one spectral band or the bandspectrum; collecting the reflected at least one spectral band or theband spectrum resulting from the one or more fluorescent targets; and,collecting at least one spectral band or the band spectrum to one ormore detector to form the image data, wherein the hyperspectral surgeryis used in surgical procedures selected from a cholecystectomy,amputation, burn, skin flap evaluation, visualizing areas ofangiogenesis, probes that bind antigens and absorbs near-infrared duringpathological evaluations and in vivo, quality control ofpharmaceuticals, monitoring vascular changes and drug discovery inresponse to pharmaceuticals, monitoring diabetic retinopathy, cancer,diabetes, sickle cell, anemia, bilirubin, raynauds, ulcers, burns, skinflaps, surgery, gallbladder, brain, monitoring wound healing, measuringoxygenation of retina, measuring optic nerve oxygenation, measuringmacular pigments, measuring pigments in retinal photoreceptors andretinal pigment epithelium, diagnosis of autoimmune retinitis, diagnosisof infectious retinitis, diagnosis of infiltrative neoplasticconditions, evaluating disease biomarkers, diagnosis of ocular traumainjuries, measuring oxygenation of a kidney and early detection of woundinfections.

Most of the art discussed above can either collect all the spectralinformation and scan the field of view, or collect all the field of viewand scan through the spectral range. Unlike the prior art, the presentinvention can collect all the spectral information and scan the field ofview simultaneously by illuminating with a spectrum, a spectroscopicwavelength, or multiple wavelengths and collect the image data, alongwith chemometrics on the source side and/or on the detector side.

One embodiment of the present invention is a hyperspectral imagingsystem having one or more optical radiation sources configured toilluminate one or more objects, illumination optics, one or moredispersing elements, one or more spatial light modulators that arecapable of tuning light from the optical radiation source into at leastone spectral band or a band spectrum (in other words, one or more singlewavelength bands or a single multi-wavelength band), and positioned inthe optical path before the one or more objects, an optical microscopeconfigured to collect the at least one spectral band or the bandspectrum reflected from the one or more objects, one or more detectors;and, a processor to process the reflected at least one spectral band orthe band spectrum to provide an enhanced image. The present inventionsystem images may also include, but are not limited to, organs andtissue components.

The optical radiation source of the system is capable of producingphotons having wavelengths in the range of 250 nm-2,500 nm. The spatiallight modulator of the system may be a Digital Micromirror Device (DMD).In certain embodiments, the dispersing element of the present system maybe a grating or a prism. The one or more detectors can also include atleast one detector selected from: a spectrometer, a two-dimensionalarray detector, a multi-array detector, an on-chip amplification CCDcamera, a back-illuminated CCD, a liquid nitrogen cooled CCD detector ora focal plane array, e.g., a CCD, VisGaAs, InGas or ferromagnetic.

In addition, the processor(s) of the present invention include, but arenot limited to, a computer having image processing software that willtune the spatial light modulator and trigger the one or more detectorsfor collecting a series of spectroscopic images formatted as ahyperspectral image cube, one or more displays, and a database ofcharacterized objects.

The present system may also have one or more liquid crystal tunablefilters positioned in the optical path, and can even be configured as aportable endoscopic system. In certain embodiments, the presentinvention can be configured for real-time in vivo analysis of the one ormore objects.

The present invention also relates to a method for obtaining spectralimage data from an object by generating a wide spectrum electromagneticradiation, separating the electromagnetic radiation into at least onespectral band or a band spectrum using a spatial light modulator,illuminating the object with the at least one spectral band or the bandspectrum electromagnetic radiations, collecting the reflected at leastone spectral band or the band spectrum electromagnetic radiationsresulting from the object, and directing the collected at least onespectral band or the band spectrum electromagnetic radiation image toone or more detector array for forming the spectral image data.

In addition, the reflected one or more single wavelength electromagneticradiations may include, but are not limited to: electromagneticradiation reflected, refracted, luminescence, fluorescence,autofluorescence, Raman scattered, transmitted, scattered, adsorbed, oremitted by the sample. In one aspect, the present invention includes oneor more slit lamps. The present invention can also operate as adissection microscope for clinical and surgical applications. In anotheraspect, the present invention includes a laparoscopic system forcreating a library of chemometric deconvolutions so that a clinician cansimply switch through various algorithms depending on applications.

In another embodiment, the present invention includes a hyperspectralimaging system including: one or more optical tunable radiation sourcesconfigured to illuminate one or more objects, e.g., tissue components;one or more detectors; one or more dispersing elements positioned in theoptical path between the one or more optical tunable radiation sourcesand the one or more detectors; one or more spatial light modulators,capable of tuning light from the optical radiation source into at leastone spectral band or a band spectrum; an optical microscope configuredto collect one or more single wavelength spectral band or a spectrumband reflected from the one or more objects positioned in the opticalpath between the optical tunable radiation sources and the one or moredetectors; one or more processors connected to the one or more opticaltunable radiation sources, the one or more detectors, the one or moredispersing element, the one or more spatial light modulator, the opticalmicroscope, or any combination thereof, wherein the one or moreprocessors process the reflected at least one spectral band or the bandspectrum as image data and to provide enhanced images. The one or moreoptical tunable radiation sources are typically capable of producingelectromagnetic radiations having wavelengths over the range of 250nm-2,500 nm and can be a supercontinuum laser.

In some aspects, the spatial light modulator is a digital micromirrordevice, and the one or more dispersing element can be a grating, aprism, a tunable filters, an electromechanical optical filter wheel, anacousto-optical tunable filter, a liquid-crystal tunable filter, adigital micromirror device, or any combination thereof. In addition, theone or more detectors can have at least one detector selected from: aspectrometer, a two-dimensional array detector, a multi-array detector,an on-chip amplification CCD camera, a back-illuminated CCD, a liquidnitrogen cooled CCD detector or a focal plane array.

In another aspect, the processor of the present invention can furtherinclude at least one processor having an image data acquisition softwarethat tunes the spatial light modulator, triggers the one or moredetectors for collection of series of spectroscopic images formatted asa hyperspectral image cube and processes the image data forvisualization; one or more transducer to digitize image data; and, oneor more display. The processor can further include a dedicated processorthat tunes the spatial light modulator, triggers the one or moredetectors for collection of series of spectroscopic images formatted asa hyperspectral image cube and processes the image data forvisualization. The processor may be further defined as a computer havingimage data processing software capable of processing various chemometricand digital signals; a digital signal processor comprising algorithmsfor chemometric analysis; and, an algorithm library for visualizingchemistry on and within the imaged one or more objects for detection,monitoring and diagnosis. In addition, the processor can be connected toa database having a library spectrum of characterized one or moreobjects. In another aspect, the one or more illumination optics of thepresent invention can include lenses, microscopes, dissectionmicroscopes, surgical microscopes, slit lamps, endoscopes, laparoscopes,colonoscopes or any combinations thereof.

In some aspects, the system of the present invention can further includeone or more liquid crystal tunable filter and can be configured as aportable endoscopic system. The present invention can also be configuredfor real-time in vivo analysis of the one or more objects.

Yet in another aspect, the present invention includes a method forobtaining spectral image data from one or more objects in vivo (e.g.,tissue components) including: generating a spectrum of electromagneticradiation; separating the electromagnetic radiation into one or moresingle wavelength bands or a spectrum band using a spatial lightmodulator; illuminating the one or more objects with the one or moresingle wavelength bands or the spectrum band; collecting the reflectedone or more single wavelength bands or the spectrum bands resulting fromthe one or more objects; and, directing the collected single wavelengthbands or the spectrum bands to one or more detectors that capturespectral image data. The method can further include the steps of: tuningthe spatial light modulator; triggering the one or more detectors forcollection; formatting and digitizing the spectral image data as ahyperspectral image; deconvoluting each pixel of the spectral image datausing a signal processing algorithm; and, displaying the spectral imageon one or more displays.

In some aspects, the method uses spectrum of electromagnetic radiationcomprises photons having wavelengths over the range of 250 nm-2,500 nm,or a supercontinuum laser. The spatial light modulator of the presentinvention can include a grating, a prism, a tunable filter, anelectromechanical optical filter wheel, an acousto-optical tunablefilter, a liquid-crystal tunable filter, a digital micromirror device,or any combination thereof. The detector for the method can have atleast one detector selected from: a spectrometer, a two-dimensionalarray detector, a multi-array detector, an on-chip amplification CCDcamera, and a back-illuminated CCD, a liquid nitrogen cooled CCDdetector or a focal plane array. It should be appreciated that thepresent invention is not limited to foregoing detectors, but may includeany type of electro-optical detection elements.

The reflected one or more single wavelength band or the spectrum bandsof the present invention can include reflected radiation, luminescence,fluorescence, autofluorescence, Raman scattered, transmitted, scattered,adsorbed or emitted electromagnetic radiation by the one or moreobjects.

In another aspect, the present invention includes a method forhyperspectral surgery by capturing image data of one or more objectgenerating a wide spectrum electromagnetic radiation including, by notlimited to the following steps: separating the electromagnetic radiationinto one or more single wavelength bands or a spectrum band using aspatial light modulator; illuminating the object with the one or moresingle wavelength bands or the spectrum band; collecting the reflectedone or more single wavelength bands or the spectrum band resulting fromthe one or more object; and, directing the collected or more singlewavelength bands or spectrum band to one or more detector to form theimage data, wherein the hyperspectral surgery is used in a surgicalprocedure selected from the group consisting of: a cholecystectomy,amputation, burn, skin flap evaluation, visualizing areas ofangiogenesis, probes that bind antigens and absorbs near-infrared duringpathological evaluations and in vivo, quality control ofpharmaceuticals, monitoring vascular changes and drug discovery inresponse to pharmaceuticals, monitoring diabetic retinopathy, diseasessuch as cancer, diabetes, sickle cell, anemia, bilirubin, raynauds,ulcers, burns, skin flaps, surgery, gallbladder, brain, monitoring woundhealing, measuring oxygenation of retina, measuring optic nerveoxygenation, measuring macular pigments, measuring pigments in retinalphotoreceptors and retinal pigment epithelium, diagnosis of autoimmuneretinitis, diagnosis of infectious retinitis, diagnosis of infiltrativeneoplastic conditions, evaluating disease biomarkers, diagnosis ofocular trauma injuries, measuring oxygenation of a kidney and earlydetection of wound infections. The method can further include the stepof covering the human subject with NIR transparent material to provideprivacy, accessibility and maintain body heat.

The present invention further includes a hyperspectral imaging systemhaving an optical path, where the system includes an illumination sourceadapted to output a light beam, the light beam illuminating a target, adispersing element arranged in the optical path and adapted to separatethe light beam into a plurality of wavelengths, a digital micromirrorarray adapted to tune the plurality of wavelengths into a spectrum, anoptical device having a detector and adapted to collect the spectrumreflected from the target and arranged in the optical path and aprocessor operatively connected to and adapted to control at least oneof: the illumination source; the dispersing element; the digitalmicromirror array; the optical device; and, the detector, the processorfurther adapted to output a hyperspectral image of the target. Thedispersing element is arranged between the illumination source and thedigital micromirror array, the digital micromirror array is arranged totransmit the spectrum to the target and the optical device is arrangedin the optical path after the target.

In some embodiments, the present invention hyperspectral imaging systemfurther includes collimating optics adapted to transmit the light beamfrom the dispersing element to the digital micromirror array as acollimated light beam. In other embodiments, the present inventionhyperspectral imaging system further includes beam shaping opticsadapted to transmit the light beam from the digital micromirror array tothe target so that the light beam substantially illuminations all of thetarget.

The present invention further includes a method of obtaining ahyperspectral image of a target including the steps of: generating abeam of light; dispersing the light beam with a dispersing element;separating the dispersed light beam into a first complex spectrum usinga spatial light modulator; illuminating the target with the firstcomplex spectrum, wherein the first complex spectrum subsequentlyreflects off of the target as a first reflected light beam; collectingthe first reflected light beam; directing the collected first reflectedlight beam to a detector to capture a first spectral image data;separating the dispersed light beam into a second complex spectrum usingthe spatial light modulator; illuminating the target with the secondcomplex spectrum, wherein the second complex spectrum subsequentlyreflects off of the target as a second reflected light beam; collectingthe second reflected light beam; directing the collected secondreflected light beam to the detector to capture a second spectral imagedata; separating the dispersed light beam into a third complex spectrumusing the spatial light modulator; illuminating the target with thethird complex spectrum, wherein the third complex spectrum subsequentlyreflects off of the target as a third reflected light beam; collectingthe third reflected light beam; directing the collected third reflectedlight beam to the detector to capture a third spectral image data; and,forming the hyperspectral image from the first, second and thirdspectral image data.

In some embodiments, the first, second and third reflected light beamseach includes reflected, luminescence, fluorescence, autofluorescence,Raman scattered, transmitted, scattered, adsorbed, or emittedelectromagnetic radiation. In other embodiments, the detector includes aprocessor having an image data acquisition software adapted to tune thespatial light modulator, trigger the detector for collection of thefirst, second and third spectral image data formatted as a hyperspectralimage cube and process the hyperspectral image cube for visualization, adigital signal process algorithm for analyzing chemometrics of thetarget and a display device adapted to display the hyperspectral image.In still other embodiments, the method is adapted for use inhyperspectral surgery, and the hyperspectral surgery is used in surgicalprocedures selected from the group consisting of: a cholecystectomy; anamputation; a burn; a skin flap evaluation; visualizing areas ofangiogenesis; probes that bind antigens and absorbs near-infrared duringpathological evaluations; in vivo quality control of pharmaceuticals;monitoring vascular changes and drug discovery in response topharmaceuticals; monitoring diabetic retinopathy and diseases such ascancer, diabetes, sickle cell, anemia, bilirubin, raynauds, ulcers,burns, skin flaps, surgery, gallbladder, brain; monitoring woundhealing; measuring oxygenation of retina; measuring optic nerveoxygenation; measuring macular pigments; measuring pigments in retinalphotoreceptors and retinal pigment epithelium; diagnosis of autoimmuneretinitis; diagnosis of infectious retinitis; diagnosis of infiltrativeneoplastic conditions; evaluating disease biomarkers; diagnosis ofocular trauma injuries; measuring oxygenation of a kidney; and, earlydetection of wound infections. In still yet other embodiments, thefirst, second and third complex spectrum each include a plurality ofintensities of a plurality of wavelengths.

These and other objects and advantages of the present invention will bereadily appreciable from the following description of preferredembodiments of the invention and from the accompanying drawings andclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The nature and mode of operation of the present invention will now bemore fully described in the following detailed description of theinvention taken with the accompanying drawing figures, in which:

FIG. 1 is a fluorescent imager for detecting, for example, ICG andbilirubin;

FIG. 2 is a short pass filter transmission characteristics used as anexcitation filter for bilirubin;

FIG. 3 is a long pass filter transmission characteristics used as anexcitation filter for bilirubin;

FIG. 4 is a diagram of a 2×2 binning process;

FIG. 5 is a block diagram depicting a method for computing the spatialresolution of the system;

FIG. 6A is a percent contrast against spatial resolution without anemission filter at the detector;

FIG. 6B is a percent contrast against spatial resolution with anemission filter at the detector;

FIG. 7 is a block diagram depicting the set up used for measurement ofabsorption spectrum of ICG;

FIG. 8 is a block diagram depicting the set up used for measuring thebest concentration of ICG in Water;

FIG. 9A is a digital image of beef fat with an ICG in water filledcapillary taken from a digital image;

FIG. 9B is an image of beef fat with an ICG in water filled capillarytaken using a present invention surgical fluorescence imager;

FIG. 10 is a SNR and contrast to background calculation wherein aprofile is taken from each row of the image and SNR and contrast tobackground are then calculated for each row and averaged;

FIG. 11A is a plot depicting the signal to noise ratio againstpenetration depth in intralipid of Aqueous ICG with constant exposure;

FIG. 11B is a plot depicting the signal to noise ratio againstpenetration depth in intralipid of Aqueous ICG with variable exposure;

FIG. 12 is a plot depicting the signal to noise ratio againstpenetration depth in intralipid of ICG with Human Bile with ConstantExposure;

FIG. 13A is a plot depicting the contrast to background ratio againstpenetration depth in intralipid of Aqueous ICG with constant exposure;

FIG. 13B is a plot depicting the contrast to background ratio againstpenetration depth in intralipid of Aqueous ICG with variable exposure;

FIG. 14 is a plot depicting the contrast to background ratio againstpenetration depth in intralipid of ICG with human bile under constantexposure;

FIG. 15 is a block diagram depicting a set up for measurement offluorescence from bilirubin;

FIG. 16 is a fluorescence image of bilirubin with excitation from 400nm-500 nm compared to a blank capillary;

FIG. 17A is a plot depicting percent contrast against spatialresolution, when no emission filter is used;

FIG. 17B is a plot depicting percent contrast against spatialresolution, when an emission filter is used;

FIG. 18 is a hyperspectral imager for use with the present invention;

FIG. 19 is an illustration of hyperspectral image cube visualization;

FIG. 20 is an embodiment of the present invention hyperspectral imagingsystem;

FIG. 21 is a diagram of an example binning in a CCD camera;

FIG. 22 is another diagram of an example binning in a CCD camera;

FIG. 23 is yet another diagram of example binnings in a CCD camera;

FIG. 24A is a graph showing a comparison of actual measured hemoglobinversus the predicted hemoglobin using pure hemoglobin samples and theactual measured hemoglobin using the present invention;

FIG. 24B is a graph showing a comparison of actual measured hemoglobinversus the predicted hemoglobin using pure hemoglobin samples and theactual measured hemoglobin using the present invention;

FIG. 25 is a schematic diagram of a microscopic system;

FIG. 26 is a diagram of the visible low resolution LCTF calibrationsetup;

FIG. 27 is a plot of visible low resolution LCTF calibration curve;

FIG. 28 is a plot of bandpass and wavelength;

FIG. 29 is a schematic of tune-wait example setup;

FIG. 30 is a graph of visible LCTF tune-wait with normalized intensityand its relationship with wavelength;

FIG. 31A is an image of a portion of the resolution target;

FIG. 31B is a graph of the reflected intensity corresponding to thetarget;

FIG. 32 is a graph of percent contrast and its relationship with spatialresolution with relay optics;

FIG. 33 is a graph of percent contrast and its relationship with spatialresolution without relay optics;

FIG. 34 is an image of color coded hyperspectral image;

FIG. 35 is a schematic of an embodiment of the present invention;

FIG. 36 is an image of oxyhemoglobin contribution taken with reflectancehyperspectral imaging system of the present invention;

FIG. 37 is an image of small vessels within a human conjuctive of an eyetaken with hyperspectral imaging system;

FIG. 38 is an example image taken on an occluded fingers using a DLP HSItwo-shot method;

FIG. 39 is a graph that shows one hundred twenty six (126) separatewavelengths using an LCTF to separate the bands prior to illumination,after which an image is captured with each bandpass or with a singleframe including up to one hundred twenty six (126) images;

FIG. 40 is a graph showing a band spectrum in which a digitalmicromirror array was used to create spectral illumination that allowsfor a lower number of images per frame;

FIG. 41 is a graph of a comparison of data obtained with the LCTF andthe digital Micromirror array illumination using the single bandwidthsof FIG. 39;

FIG. 42 is a graph of a comparison of data obtained with the LCTF andthe digital Micromirror array illumination using the single bandwidthsof FIG. 40;

FIG. 43 is a flow chart of the basic 2-shot algorithm;

FIG. 44 is a flow chart of the processing of the data cube obtainedusing the basic 2-shot method;

FIG. 45 is a flow chart of the acquisition method of the basic 2-shotmethod;

FIG. 46 is a sample of images obtained for a finger occlusion atdifferent times using the present invention;

FIG. 47 is a comparison of images obtained using visible light and Nearinfrared (NIR) of the reperfusion of a foot following removal of theshoe;

FIG. 48 is the in vivo hyperspectral imaging of human tissue, spatialvariation of percentage of HbO₂ and surface temperature in response toburn;

FIG. 49 is a schematic illustration of the different elements of a DLP®hyperspectral imaging system of the present invention;

FIG. 50A is the normalized absorption spectra of HbO₂ and Hb used asreferences for multivariate deconvolution of spectral sweep absorbancecubes;

FIG. 50B is the positive and negative subtraction of spectra used for “3shot” illumination;

FIG. 51 is a block diagram that shows the experimental procedure used tocapture images using the “3 shot” method of the present invention andthe MATLAB® algorithm used to process the images captured using the “3shot” method of the present invention;

FIG. 52A is an illustration of the “3 shot” illumination method of thepresent invention used for visualizing blood oxygenation showing thenormalized absorbance spectra in the 520 nm-645 nm wavelength range forHbO₂ and Hb subtracted from each other, and the positive areas becomethe two illumination spectra;

FIG. 52B is an illustration of the “3 shot” illumination method of thepresent invention used for visualizing blood oxygenation showingrelative intensity of each illumination spectrum is stretched from 0 to100 to maximize the overall light intensity and match the requiredOL-490 input format;

FIG. 52C is an illustration of the “3 shot” illumination method of thepresent invention used for visualizing blood oxygenation showing theabsorbers and scatterers that are not HbO₂ and Hb;

FIG. 53A is a graph illustrating the comparison of optical outputmeasured by a spectrometer and desired optical output for firstillumination spectrum of the “3 shot” method before applying centerwavelength calibration and intensity adjustment;

FIG. 53B is a graph illustrating the comparison of optical outputmeasured by a spectrometer and desired optical output for firstillumination spectrum of the “3 shot” method after applying centerwavelength calibration and intensity adjustment;

FIG. 54 is a timing diagram of hyperspectral acquisition where N is thetotal number of slices to be acquired, for a spectral sweep, N=126, andfor a “3 shot”, N=3;

FIG. 55A is a diagram illustrating the processing algorithms forvisualizing the images of blood oxygenation captured using the spectralsweep method comparing measured spectrum to reference spectra viamultivariate least squares analysis to quantify relative concentrationHbO₂;

FIG. 55B is a diagram illustrating the processing algorithms forvisualizing the images of blood oxygenation captured using the “3 shot”method of the present invention which subtracts the image representingHb absorbance from the image representing HbO₂ absorbance and dividingthe broadband absorbance to quantify relative concentration of HbO₂;

FIG. 56A is the visualization of an ischemia induced by occluding bloodflow to a finger imaged with the DLP HSI in spectral sweep mode;

FIG. 56B is the visualization of an ischemia induced by occluding bloodflow to a finger imaged with the DLP HSI in “3 shot” mode;

FIG. 57A is the average of five spectral sweep oxyz output imagescaptured as ‘Control’;

FIG. 57B is the average of five spectral sweep oxyz output imagescaptured while ‘Occluded’;

FIG. 57C is the average of five spectral sweep oxyz output imagescaptured for ‘Reperfusion’;

FIG. 58A is a color-coded “3 shot” output images captured immediatelybefore cutting the rubber band;

FIG. 58B is a color-coded “3 shot” output images captured 10 secondsafter cutting the rubber band;

FIG. 58C is a color-coded “3 shot” output images captured 2 minutesafter cutting the rubber band;

FIG. 59 is the real-time progression of reactive hyperemia after cuttingrubber band from finger shown as a plot of average pixel values from “3shot” images acquired during removal of tourniquet from finger whichshows reactive hyperemia in at 3 frames per second;

FIG. 60 is an image of a hyperspectral imaging setup in animal labsurgical suite at UTSW for imaging porcine partial nephrectomy;

FIG. 61A is an image of a kidney showing a ‘Control’;

FIG. 61B is an image of a kidney showing ‘Occluded’;

FIG. 61C is a spectral sweep image of a kidney showing a ‘Control’;

FIG. 61D is a spectral sweep image of a kidney showing ‘Occluded’;

FIG. 61E is a “3 Shot” image of a kidney showing a ‘Control’;

FIG. 61F is a “3 Shot” image of a kidney showing ‘Occluded’;

FIG. 62 is a graph plotting monitoring the time progression percentageof oxyhemoglobin perfusing the kidney before and during renal AO and AVocclusion;

FIG. 63A is a raw image of a bilateral ischemia exhibited in a pigkidney having two renal arteries taken immediately after clamping onerenal artery;

FIG. 63B is an image of a bilateral ischemia exhibited in a pig kidneyhaving two renal arteries taken with the “3 shot” method after clampingone renal artery showing that the lower pole is ischemic, but the upperpole is still highly oxygenated;

FIG. 63C is an image of a bilateral ischemia exhibited in a pig kidneyhaving two renal arteries taken with the “3 shot” method after clampingthe second renal artery showing that the upper pole becomes ischemic aswell;

FIG. 64A is a raw image of a human kidney having partial nephrectomy;

FIG. 64B is an image of a human kidney having partial nephrectomy takenusing the “3 Shot” method before clamping;

FIG. 64C is an image of a human kidney having partial nephrectomy takenusing the “3 Shot” method immediately after clamping;

FIG. 64D is an image of a human kidney having partial nephrectomy takenusing the “3 Shot” method after the tumor is removed and still clamped;

FIG. 64E is an image of a human kidney having partial nephrectomy takenusing the “3 Shot” method immediately after clamping;

FIG. 64F is an image of a human kidney having partial nephrectomy takenusing the “3 Shot” method after dissection;

FIG. 65 is a graph showing the progression of relative oxygenationmeasured from output images in FIGS. 23B-23E;

FIG. 66 is a digital image of a full leg with a superimposed spectralsweep image mapping the surface oxygenation of a patient with lower limbneuropathy;

FIG. 67 is a graph showing the spatial progression of surfaceoxygenation as measured along the centerline of the leg in thehyperspectral images in FIG. 66 wherein the dashed line represents thelimit of neuropathy as determined by a physician administering aclinical nervous response exam on the leg;

FIG. 68A is a control image of a rabbit brain taken with the “3 shot”method with the black sample area showing normal tissue and blue samplearea showing damaged tissue;

FIG. 68B is an image of a rabbit brain taken with the “3 shot” methodafter oxygen supply is cut off to the rabbit wherein the normal tissuebecomes ischemic and there is no change in the damaged tissue;

FIG. 69A is an image of a human hand having burned regions thereon;

FIG. 69B is a hyperspectral image of the human hand of FIG. 69Achemically encoded to show oxyhemoglobin;

FIG. 69C is a hyperspectral image of the human hand of FIG. 69Achemically encoded to show water;

FIG. 70A is an representation of a portion of a digital micromirrordevice showing a single column of micromirrors in an “on” position;

FIG. 70B is a representation of an illumination spectrum produced by thedigital micromirror device of FIG. 70A;

FIG. 71A is an representation of a portion of a digital micromirrordevice showing a single column of micromirrors in an “on” position;

FIG. 71B is a representation of an illumination spectrum produced by thedigital micromirror device of FIG. 71A;

FIG. 72A is an representation of a portion of a digital micromirrordevice showing a plurality of columns of micromirrors in an “on”position where respective columns have different numbers of micromirrorsin an “on” position; and,

FIG. 72B is a representation of a complex illumination spectrum producedby the digital micromirror device of FIG. 72A.

DETAILED DESCRIPTION OF THE INVENTION

At the outset, it should be appreciated that like drawing numbers ondifferent drawing views identify identical, or functionally similar,structural elements of the invention. While the present invention isdescribed with respect to what is presently considered to be thepreferred aspects, it is to be understood that the invention as claimedis not limited to the disclosed aspects.

Furthermore, it is understood that this invention is not limited to theparticular methodology, materials and modifications described and assuch may, of course, vary. It is also understood that the terminologyused herein is for the purpose of describing particular aspects only,and is not intended to limit the scope of the present invention, whichis limited only by the appended claims.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood to one of ordinary skill inthe art to which this invention belongs. As used herein, DLP® andDigital Micromirror Device (DMD) are used interchangeably. A DMD chiphas on its surface several hundred thousand microscopic mirrors arrangedin a rectangular array which correspond to the pixels in the image to bedisplayed. The mirrors can be individually rotated ±10-12°, to an on oroff state. In the on state, light from the projector bulb is reflectedinto the lens making the pixel appear bright on the screen. In the offstate, the light is directed elsewhere (usually onto a heat sink),making the pixel appear dark. It can be purchased commercially fromTexas Instruments. As used herein, “hyperspectral” in reference to animaging process or method is meant the acquisition of an image at morethan one wavelength or bands of wavelengths. Furthermore, as usedherein, “spectra” is defined as illumination with spectra of light withmultiple wavelengths within a spectral range. Although any methods,devices or materials similar or equivalent to those described herein canbe used in the practice or testing of the invention, the preferredmethods, devices, and materials are now described. Still further, asused herein, “hyperspectral image” is defined as an image derived fromhyperspectral image data, e.g., hyperspectral data cube, which includesbut is not limited to, a chemically encoded image, a principle componentanalysis (PCA) image and wavelength dependent images.

A spectral illuminator using DLP technology may be used to excitefluorescence markers, which are native to the tissue (tissuefluorescence) or target, or markers that were injected or painted ontothe tissue or target. An optical filter, Liquid Crystal Tunable Filter(LCTF) or other element can be used to pass emitted fluorescence lightto a detector. For imaging bile ducts, e.g., ICG, indocyanine green canbe injected or the native fluorescence properties of bile can be used.With improved imaging, surgical procedures can focus on the target fortreatment (or the adjacent tissue) by visualization, e.g., the biliarytree. The system can be used for other surgical applications to performin vivo pathology during surgery, i.e., in vivo hyperspectral pathology.The present invention can also be used in other methods or techniques,including but not limited to, in vivo pathology of cancer in real-time(e.g., before, during or after surgery), in the body (e.g.,laparoscopically), without having to biopsy and/or fix the tissue ortarget to a slide. The example being a probe that binds to antigens. Thetechniques described infra are also applicable to fluorescencemicroscopy. Finally, chemometrics and digital signal processing may beused to produce chemically encoded images to enhance visualizations.Furthermore, it has been found that using a digital light processor(DLP), the present inventors were able to achieve video ratehyperspectral image capture and processing. Such video rate capture andprocessing is not possible with current illumination technology;however, is made possible by the use of DLP technology as set forthherein. The video capture rate also benefits from the chemometricillumination.

In addition to the above described disadvantages of present imagingapparatus and techniques, the following describes still furtherdisadvantages. ERC is an invasive, user-dependent modality which mayalso induce pancreatitis. Though ultrasound is cost effective, theimages are not clearly understood by clinicians. MRC is also ofteninconclusive in patients with air in the biliary system. Proponents ofroutine intraoperative cholangiography (IOC) claim that this practicelowers risk of CBD injuries and leads to fewer retained bile ductstones. (See Flowers J L, Zucker K L, Graham S M, et al. “Laparoscopiccholangiography: results and indications.” Ann Surg 1992; 215:209-16;and, Gerber A, Apt MK. “The case against routine operativecholangiography. AM J Surg” 1982; 143:734-6). The disadvantages ofroutine IOC include increased costs, operative time and false positivefindings, leading to unnecessary efforts to clear the CBD stones.Injecting contrast media for a CTC can led to adverse effects such asanaphylaxis, urticaria and respiratory distress. (See A Persson, NDahlstrom, Ö Smedby and T B Brismar: “Three-dimensional drip infusion CTcholangiography in patients with suspected obstructive biliary disease:a retrospective analysis of feasibility and adverse reaction to contrastmaterial”). These can be reduced if the contrast agents are diffusedinstead of injected. Possible explanations of infrequent use of CTCmight be the low resolution of a single detector helical CT and reportsof an unacceptable high number of adverse events after the injection ofmeglumine iotroxate. (See A Persson, N Dahlstrom, Ö Smedby and T BBrismar: “Three-dimensional drip infusion CT cholangiography in patientswith suspected obstructive biliary disease: a retrospective analysis offeasibility and adverse reaction to contrast material”). It has beenobserved that selective preoperative ERCP was successful in showing CBDstones.

Fluorescence is the optical phenomenon of luminescence in which themolecular absorption of a photon triggers the emission of another photonof longer wavelength. The energy difference between the absorbed and theemitted wavelength ends up as molecular vibration or heat. A materialthat exhibits fluorescence is called a fluorophore. Differentfluorophores have different absorption and emission wavelengths.Fluorescence imaging has found a number of applications in the field ofbiochemistry and medicine. Typically a fluorophore molecule, in theground state S₀, absorbs energy hσ provided by the excitation light.This energy takes the molecule to an excited state S₁. This is anunstable state; hence, the molecule returns to the ground state S₀ byemitting energy equivalent to hν. The return to the ground state canhappen through many paths.

In the current research Indocyanine Green (ICG) and bilirubin are usedas fluorophores. ICG has already been established as an injectiblefluorophore for retinal angiography. Bilirubin on the other hand existsas a component of bile, thus, avoiding injection of an additionalfluorophore.

One of the major hurdles during a cholecystectomy procedure, open orlaparoscopic, is the visualization of the bile duct, which is hiddenunderneath fat layers, to avoid any bile duct injuries. As discussedabove, the known imaging techniques are not fully satisfactory.

As described supra, the critical issue while performing aCholecystectomy is injury to the Common Bile Duct (CBD) which can causepost procedural complications. Since the CBD is under a fat layer it isimportant to image and locate the CBD during the procedure. The presentinvention fluorescence imager helps visualize the CBD by using thefluorescence properties of Indocyanine Green, which is injected into thebody, or by using the fluorescence properties of bilirubin, alreadyexisting in bile. An instrument using ICG as a fluorophore has beencharacterized for its penetration depth to view fluorescence using anintralipid model, thereby determining the best possible fluorescenceconcentration that would give maximum fluorescence photons at the sametime remaining within the limits of the concentration prescribed forhuman dosage. The system incorporates unique short pass-long pass filtercombinations at the source and detector ends, respectively, to provideexcitation illumination and detect emission. The filters are designed inaccordance with the absorption and emission characteristics ofindividual fluorophores in different mediums.

The best fluorescence concentration was found to be 0.015 mg/ml. Depthanalysis was performed for ICG mixed with water, going deeper in a 1%intralipid solution used as a model to mimic tissue and fat properties.A vernier height gauge was coupled to a capillary holder, which held thecapillary containing the fluorophore. Two separate analyses were carriedout, ICG mixed with human bile and aqueous ICG solution, both having anapproximate concentration of 0.015 mg/ml. Contrast to background andsignal to noise ratios were computed at each depth to find the maximumdepth the system can visualize. The maximum depth of penetration wasfound to be 11 mm below the surface of the intralipid solution when ICGwas mixed with bile and 20 mm below the surface of the intralipidsolution for aqueous ICG. The threshold for contrast to background wasset based on beef fat measurement.

Thus, the present invention includes devices and methods forfluorescence imaging that allows the user to visualize the bile ducteither by injection of a fluorophore into the bile duct or by excitingthe fluorophore inherently present in the bile duct. As described above,this can be accomplished by the use of fluorophores either existinginside the bile duct (bilirubin) or through external injection into thebile duct (Indocyanine Green). A unique design of short pass-long passfilters enables viewing all emitted photons due to fluorescence. Theexcitation fluorescence filter is coupled to a broadband QTH source,which provides the energy for excitation of the fluorophore. Thefluorophore absorbs this energy and emits photons of a higher wavelengththan the one being absorbed. These photons are detected by the FPA whichis coupled to the emission fluorescence filter, designed uniquely toprevent the transmission of any excitation light. The FPA produces animage of the fluorescence of the fluorophore, which is then used forfurther analysis and to determine the location of the bile duct.

The present invention includes a novel fluorescence imagery system forvisualizing the biliary tract, in-vivo, using Indocyanine Green andBilirubin as fluorophores. The present invention imaging system includesa focal plane array (FPA) sensitive in the near infrared region forimaging Indocyanine green (ICG). The present invention imaging systemhelps surgeons visualize the bile duct real-time by exploiting thefluorescence properties of ICG.

The system includes of a broadband illumination source of light from a250 W Quartz Tungsten Halogen (QTH) lamp placed in a housing (Oriel,Stratford, Conn.). The source is powered by a radiometric power supplyto maintain a stable lamp output with minimum light ripple, making it anexcellent long term, stable illuminator. The condensing lens assemblyincludes a molded Pyrex® aspheric which focuses light through thefilters and to the optical couplers. A liquid light guide (Oriel,Stratford, Conn.), which also acts as an UV filter, i.e., attenuateswavelengths below 420 nm, is coupled to the condenser. The other end ofliquid light guide is coupled to a Pyrex® aspheric expander whichconverts a fiber optic light beam to a collimated light beam. Thebroadband illumination is attenuated by a low pass filter mounted infront of the aspheric expander. The low pass filter (Omega Optical,Brattleboro, Vt.) was selected taking the ICG absorption characteristicsinto account. The lowpass filter (LPF) has a cut off (50% Transmittance)wavelength at 795 nm. The detector includes a high pass filter (HPF)coupled to a 50 mm, f/1.4 lens (Nikon, Tokyo, Japan) which is placed infront of the focal plane array.

The foregoing embodiment is represented in FIG. 1. Lamp 100 transmitslight through collector optics 102 and into light guide 104. The lightis then transmitted from light guide 104 through beam shaping optics 106and filter 108. After exiting filter 108, the light reflects off ofsurface 110 and the reflected light is subsequently passes throughfilter 112 and lens 114 which in turn transmits the light to camera 116.

The high pass filter (Omega Optical, Brattleboro, Vt.) was selectedaccording to the emission characteristics of ICG. The HPF has a cut off(50% Transmittance) wavelength at 805 nm. The CCD includes a Focal Planearray. The system utilizes a PIXIS 400 BR (Princeton Instruments,Trenton, N.J.). The PIXIS 400 BR FPA is a fully integrated system with apermanent vacuum and deep cooling arrangement. It uses a highperformance, back illuminated, spectroscopic format CCD. The CCDincorporates deep depletion technology to extend the sensitivity in thenear infrared (NIR). These special devices are thermoelectrically cooled(air) down to −75° C. to provide the lowest dark charge and therebyreduction of dark current noise. The FPA has a 1340×400 pixel arrayhaving an 8 mm chip height and 27 mm spectral coverage. This arrangementmakes the FPA ideal for multistripe spectroscopy and maximum lightcollecting area. Each pixel has a size of 20 μm×20 μm with a totalimaging area of 26.8 mm×8.0 mm. The device's sensitivity ranges from 220nm to 1100 nm, with a peak efficiency of about 85% at 800 nm. Thisdevice has a 16 bit Analog to Digital Converter (ADC) coupled to the FPAwhich provides a maximum of 2¹⁶ (655356) shades of gray. A high endlaptop (e.g., Dell Latitude D610, Austin, Tex.) is connected to thecamera for image rendering and analysis. The foregoing example system ishelpful for a surgeon to visualize the common bile dust duringcholecystectomy; however, this system is presented only as an exampleand does not limit the scope of the claimed invention.

The fluorescence imaging system using bilirubin as a fluorophoreincludes a Focal Plane Array which is sensitive in the visible region.The system also includes a broadband illumination of light from a 250 WQuartz Tungsten Halogen (QTH) lamp placed in a housing (Oriel,Stratford, Conn.). The source is powered by a radiometric power supplyto maintain a stable lamp output with minimum light ripple, making it anexcellent long term, stable illuminator. The condensing lens assemblycontains a molded Pyrex® aspheric which focuses light toward the filtersand optical couplers. A liquid light guide (Oriel, Stratford, Conn.),which also acts as an UV filter, i.e., attenuates wavelengths below 420nm, is coupled to the condenser. The other end of the liquid light guideis connected to a beam expander/collimator. A low pass filter (OmegaOptical, Brattleboro, Vt.) with a cut off (50% Transmission) at 500 nm,selected on the basis of the absorption properties of bilirubin, iscoupled to the collimating assembly. A high pass filter (Omega Optical,Brattleboro, Vt.) with a cut off (50% Transmission) at 515 nm, selectedon the basis of the emission properties of bilirubin, is coupled to a 50mm, f/1.4 lens (Nikon, Tokyo, Japan) which is placed in front of thefocal plane array.

The present invention fluorescence system includes CoolSnap_(ES)(Photometrics, Tucson, Ariz.) which contains a Sony ICX 285 focal planearray. Higher sensitivity is one of the distinct advantages of this CCDwhich enables it to have a reduced time for data collection. The analogto digital converter (ADC) is 12 bits with a speed of 20 Mpixels/s,thereby decreasing the acquisition time by a factor of 20 whendigitizing the same number of pixels.

Again, the foregoing embodiment is represented in FIG. 1. Lamp 100transmits light through collector optics 102 and into light guide 104.The light is then transmitted from light guide 104 through beam shapingoptics 106 and filter 108. After exiting filter 108, the light reflectsoff of surface 110 and the reflected light is subsequently passesthrough filter 112 and lens 114 which in turn transmits the light tocamera 116.

The specific system components described infra are for illustrativepurposes only and are not intended to limit the scope of the claimedinvention. The QTH source (Lamp and Housing) along with the radiometricpower supply are manufactured by Spectra-Physics, a division of NewportCorporation. The research Grade lamp housing (Model: 66884) houses a 250W QTH lamp source (Model: 6334) and the Radiometric power supply (Model:69931) drives the source.

The Oriel Radiometric Power Supply is a highly regulated source ofconstant current or constant power for QTH lamps. The control andmonitoring features of the power supply include powering the supply onand off, set the current/power preset and limit, monitor the current,voltage, power and operating time. The power supply is run in currentmode with a current setting at 10.42 Amps, which drives the QTH lamp.(See ‘The Newport Resource’ 2008-2009. Newport Corporation pp 140-141).The Research grade lamp housing holds the QTH lamp. The housing includescondensing optics to produce a collimated or focused beam. Aspherecondensers are used for superior uniformity. The housing alsoincorporates rear reflectors to collect the lamp's back radiation,external lamp and reflector adjustments to fine position the filamentand a power regulated fan to cool the lamp and the housing. (See ‘TheNewport Resource’ 2008-2009. Newport Corporation pp 137-138). Re-imagingonto the filament does increase the collimated output a little andchanges the power balance of the system.

TABLE 1 Specification of Radiometric Power Supply (Model 69931)Parameters Model 69931 Power Factor >0.99 Input Voltage 90-264 VAC InputCurrent 5 A Input Frequency 47-63 Hz Output Power 40-300 W OutputCurrent 3-24 A Output Voltage Range 0-45 VDC Line Regulation 0.01%Output Voltage Ripple <0.05% r.m.s. Light Ripple <0.05% r.m.s. MeterAccuracy (% of full <0.05% scale) Digital Meter Resolution, 0.1 VDCVoltage Digital Meter Resolution, 0.01 A Current Digital MeterResolution, 1 W Power Safety Interlock Voltage 12 VDC/GND Operating ModeConstant current or constant power Ambient Operating 0-45° C.Temperature Weight 20 (9) Dimensions (W × D × H) 12.0 × 16.0 × 5.18 (305× 406 × [in. (mm)] 132)

Quartz Tungsten Halogen lamps are popular visible and near infraredsources because of their smooth spectral curve and stable output. ICGhas a molecular formula of C₄₃H₄₇N₂NaO₆S₂. The filter designs have to beselected based on the emission and absorption spectrum of ICG. ICG has abimodal absorption spectrum from 650 nm-800 nm, with peaks occurring at685 nm and 775 nm in distilled water. In plasma, ICG has two absorptionpeaks at around 710 nm and 805 nm. The absorption spectrum changesaccording to the concentration of ICG and solvent. For example, higherconcentrations have a maximum absorption at 685 nm, while lowerconcentrations have a maximum absorption at 775 nm in an aqueoussolution. (See M. L. J. Landsman, G. Kwant, G. A. Mook, W. G. Zijlstra,“Light-Absorbing Properties, Stability and Spectral Stabilization ofIndocyanine Green”, Jour. Applied Physiology, Vol. 40, No. 4, April1978). Thus, separate filters were designed for ICG with water as asolvent and ICG with blood as a solvent. Instead of a conventional bandpass filter combination for emission and excitation filters, unique lowpass and high pass filters are used as excitation and emission filters.

The filter combination (excitation and emission) of low pass and highpass filters are used. The low pass filter has a cut off (50%Transmission) wavelength at 790 nm. This filter coupled with the source(mentioned above) is used to excite ICG with all possible absorptionwavelengths of ICG including the peak absorption wavelength. The highpass filter (coupled with the detector) was used to collect thefluorescence photons coming from ICG. In an aqueous solution ICG, has anemission peak at 820 nm. (See R. C. Benson, H. A. Kues, “Fluorescenceproperties of Indocyanine Green as Related to Angiography”, Vol. 23, No.1, 159-163, Phys. Med. Biol. 1978). Thus, a high pass emission filterwith a cut-off (50% Transmission) at 805 nm is used. This enables us toavoid any excitation light reflected back. These curves were obtained byusing a standard calibrated USB2000+ spectrometer (Ocean Optics,Dunedin, Fla.) and using the filter in the transmission mode andilluminating with the broad band source discussed above. During theentire study all the parameters including distance, exposure time andother pre-processing remained the same.

ICG in blood follows a similar pattern as that for ICG in an aqueoussolution with changes in the Excitation and Emission cut-off filters.The low pass excitation filter has a wavelength cut off (50%Transmission) at 810 nm. This filter coupled with the source enablesilluminating the target (ICG) with its entire absorption wavelength. ICGemission occurs at 830 nm when in blood. (See R. C. Benson, H. A. Kues,“Fluorescence properties of Indocyanine Green as Related toAngiography”, Vol. 23, No. 1, 159-163, Phys. Med. Biol. 1978). Thus, thehigh pass emission filter has a filter cut-off (50% Transmission) at 815nm allowing the viewing of only the fluorescence photons coming into thedetector.

Bilirubin stands as one of the important constituents of bile. The pH ofbile ranges from 7.5-9.5, which indicates that bile is alkaline. This isdue to the presence of bicarbonates. Bilirubin is a major product ofheme catabolism. Bilirubin is a yellow tetrapyrrole pigment which iswater soluble as it possesses two propionic side chains, which might beexpected to render it highly polar. Bilirubin may be water insoluble asthe bilirubin molecule can adapt to various configurations. Bilirubin ismostly found in the conjugated form in bile. Unconjugated bilirubin isnormally 5% of total bilirubin. Bilirubins are also susceptible tooxidation and are photosensitive thereby leading to a variety ofderivatives. (See Francesco Baldini, Paolo Bechi, Fabio Cianchi, AlidaFalai, Claudia Fiorillo, Paolo Nassi “Analysis of Optical Properties ofBile” J. Biomedical Optics 5(3), 321-329, July 2000). The quantum yieldof free bilirubin at room temperature is low (<10⁻⁴). (See M. A. Rosci,“Fluorescence of free bilirubin at room temperature”, Experimentia Vol:39 (1983)). Thus, fluorescence of free bilirubin is difficult toobserve. However fluorescence of bilirubin is enhanced in the presenceof albumin. (See Humra Athar, Nisar Ahmad, Saad Tayyab, Mohammad A.Qasim “Use of Fluorescence enhancement technique to studybilirubin-albumin interaction” Int. J. Biological Macromolecules, 25,353-358, 1999).

Bilirubin in bile fluoresces as a result of its interactions with otherbile components and albumin. Thus, imaging this fluorescence is anindicator for the position of bile duct, which contains bile in it.Bilirubin has a strong absorption of visible light between 400 nm-500nm. (See Francesco Baldini, Paolo Bechi, Fabio Cianchi, Alida Falai,Claudia Fiorillo, Paolo Nassi “Analysis of Optical Properties of Bile”J. Biomedical Optics 5(3), 321-329, July 2000). Thus, a filter designedto allow ideally all the wavelengths below 500 nm would become anexcellent excitation filter for fluorescence of bilirubin in bile. Thiswas the basis of the excitation filter selection in the present example.The transmission characteristics of such a filter are shown in FIG. 2.The fluorescence of bilirubin in bile is observed to peak around 528 nm.(See Humra Athar, Nisar Ahmad, Saad Tayyab, Mohammad A. Qasim “Use ofFluorescence enhancement technique to study bilirubin-albumininteraction” Int. J. Biological Macromolecules, 25, 353-358, 1999).Thus, a filter capable of eliminating the excitation wavelengths andable to measure or see the fluorescence was selected. Such a filter wasimplemented through a long pass filter with its cut off (50%) at 515 nm.The transmission of such a filter is shown in FIG. 3.

The emission filter is coupled to a 50 mm, f/1.4 Nikon Lens manufacturedby Nikon. This lens helps in focusing the target onto the CCD. This lensis fast enough for shooting virtually all types of light. It producesdistortion free images with high resolution and color rendition. Thislens incorporates a wide variety of f-stops or apertures. The maximumaperture is f/1.4, while the minimum aperture is f/16. The back end ofthe lens is coupled to a C-F mount, which is used to fit a variety oflenses onto the CCD.

The Charged Coupled Device or Focal Plane Array is an analog shiftregister that enables analog signals to be transported throughsuccessive stages controlled by a clock signal. An image is projected bya lens onto the photoactive region of the CCD which includes photodiodesand capacitors that accumulate charges based on the intensity of lightat that location. This image is then transferred to the transmissionregion, which includes shift registers. Once the array has been exposedto the image, a control circuit causes each capacitor to transfer itscontent to its neighbor. The last capacitor dumps the charge to a chargeamplifier, which converts the charge into a voltage and is finally readout.

The Focal Plane Array used for imaging Indocyanine Green (ICG) is thePIXIS 400 BR. The specifications for PIXIS 400BR is given in Table 2below.

TABLE 2 Specification for PIXIS 400 BR Specification PIXIS 400 BR CCDFormat 1340 × 400 imaging pixels 20 × 20-μm pixels 100% fill factor 26.8× 8.0-mm imaging area System Read Noise 5 e-rms @ 100-kHz digitzation(max) 16 e-rms @ 2 MHz digitization (max) Spectrometric Well 300ke-(High Sensitivity-Typical) Capacity 1 Me-(High Capacity-Typical)Deepest Cooling −70° C. (High Sensitivity) Minimum Temperature −75° C.(High Sensitivity) Typical Dark Current @ 0.25 e-/p/s (Typical) HighSensitivity −75° C. 0.5 e-/p/s (Maximum) Dynamic Range 16 Bits VerticalShift Rate 30 μsec per row Operating +5 to +30° C. non-condensingEnvironment

PIXIS 400 BR has controlling software that controls and maintains theset temperature allowing a deviation of ±0.05° C. from the settemperature by controlling the camera's cooling circuit. Dark charge isthermally induced into the FPA over time. This statistical noise iscalled the dark noise. Dark noise tends to change depending on theexposure time, temperature and gain. Dark noise could be measured whenthere is no light passing through the FPA. The longer the exposure timeand warmer the temperature, the background becomes larger and lessuniform. (See PIXIS User Manual). The camera has controller gainsoftware which allows setting three different gain levels. Level 1 (low)is used for high signal intensities. Level 2 (mid) is used for midintensity levels and Level 3 (high) is used for low intensities.

From the specification table one can see that the camera has a dualdigitization rate available (100 KHz/2 MHz). The 2 MHz digitization rateis used for the fastest possible data collection, while the 100 KHz isused when noise performance is of greatest concern. Thus, multidigitization allows complete freedom between ‘Slow Operation’ for lownoise and high SNR and ‘Fast Operation’ for rapid spectral acquisition.(See PIXIS User Manual).

Megapixel resolution and smaller pixel size allows imaging very finedetails. Sensitivity can be improved by binning, but at the expense ofresolution. The camera has a 1340×400 pixel CCD array that providessuperior resolution over the industry standard ‘1024’ pixel format.Binning increases the frame rate. The camera has a dynamic range of 16bits allowing bright and dim signals to be quantified in a single imagewith 216 shades of gray.

The PIXIS 400 BR has a back illuminated FPA. This means that lightenters from a back surface through a thinned (etched) silicon layer.This has an advantage that no light absorption or reflection takes placeat the polysilicon gate structure enabling the CCD to have higherquantum efficiency (almost twice the efficiency). Due to this etching,the layer becomes transparent to NIR wavelengths causing fringe effectsfor NIR spectroscopy. To overcome the disadvantages of etaloning, theCCD is made of thicker silicon (roughly twice the thickness of a normalback-illuminated CCD). This contributes significantly to the absorptionof NIR light, reducing the amount of light that survives a round trippath to cause interference and increasing the quantum efficiency. Thisreduces the amount of light into the CCD that is reflected back from thepolysilicon side of the back surface. It also increases the quantumefficiency by increasing the amount of light into the CCD and reducingstray light in the spectrometer.

The CoolSnap_(ES) camera is used as a detector for measuring thefluorescence of bilirubin. This camera has high quantum efficiency inthe visible region which makes it a perfect choice for fluorescencemeasure of bilirubin. This camera is used in applications requiring highspeed and high spatial resolution in the visible region. It ismanufactured by Roper Scientific (Now Princeton Instruments). TheCoolSnap_(ES) offers higher sensitivity and lower read noise to producehigh quality 12-bit monochrome images. (See CoolSnap_(ES) user manual).Exemplary specifications are provided is Table 3 (below).

TABLE 3 Specifications of CoolSnap_(ES) FPA Specification CoolSnap_(ES)CCD Format 1392 × 1040 imaging pixels 6.45 × 6.45-μm pixels 8.77 ×6.6-mm imaging area (Optically Centered) System Read Noise <8 e-rms @ 20MHz Well Capacity 16000 e-(Single Pixel) 30000 e-(2 × 2 Binned Pixel)Cooling Thermoelectric, 5° C. below ambient Temperature Dark Current 1e-/p/s Dynamic Range 12 Bits @ 20 MHz Dimensions 4.5″ × 5.0″ × 2.5″ (1.9lbs) Operating Environment 15° C. to 30° C. ambient Frame readout 91ms/frame

The CoolSnap_(ES) incorporates a SONY ICX285AL silicon chip array withinterline transfer capability. The interline transfer CCD has a parallelregister that is subdivided into alternate columns of sensor and storageareas. The image accumulates in the exposed area of the parallelregister and during CCD readout the entire image is shifted underinterline mask into a hidden shift register and then proceeds in normalCCD fashion. Since the signal is transferred in microseconds, smearingis undetectable for typical exposures. However, a drawback to interlinetransfer CCD's has been their relatively poor sensitivity to photonssince a large portion of each pixel is covered by the opaque mask. As away to increase the detector's fill factor, high quality interlinetransfer devices have microlenses that direct the light from a largerarea down to the photodiode. The quantum efficiency is about 60% in theregion of bilirubin-albumin fluorescence (500 nm-550 nm), hence theseCCDs are used as detectors for measuring the fluorescence ofbilirubin-albumin.

Both the above mentioned detectors are driven by Photometrics VirtualCamera Access Method (PVCAM) software. The PVCAM application programminginterface for high performance digital cameras is a set of softwarelibrary routines that implement a camera's operations in a hardwareindependent, platform independent suite of function calls. This softwareis used to control and acquire data from the camera. The data collectionis done using Vpascal programming integrated in V++ which internallycommunicates with PVCAM for controlling the detectors.

Binning is a process of combining charge from adjacent pixels in a CCDduring readout. This process is performed prior to digitization in theon-chip circuitry of the CCD by specialized control of the serial andparallel registers. The two primary benefits of binning are improvedsignal-to-noise ratio (SNR) and the ability to increase frame rate, atthe expense of spatial resolution. Binning 1×1 has the maximal spatialresolution. No charges are combined in this case. In the case of 2×2binning, during parallel readout, the charges from two rows of pixels,rather than a single row, is shifted into the serial register. Next,charge is shifted from the serial register, two pixels at a time, intothe summing well. It then goes to the output amplifier. This process isiterated until the entire array is read. One of the prime advantages ofbinning is high SNR. In normal operation, CCD read noise will be addedto each pixel, whereas during binning, the CCD read noise is added toeach super pixel thereby increasing the SNR. A 2×2 binning process isshown in the FIG. 4.

The following characterization of indocyanine green fluorescence imageris provided for clarification. Indocyanine green (ICG) (C₄₃H₄₇N₂O₆S₂Na),having a molecular weight of 775, is a trycarbocyanine type of dye withinfrared absorbing properties. ICG has little or no absorption in thevisible spectrum. ICG may be used for recording dye dilution curves, inparticular for determination of cardiac output. The principle advantagesof using ICG as a dye are its absorption maximum at the isobestic pointof hemoglobin and oxy-hemoglobin, the confinement to the vascularcompartment through plasma protein binding, low toxicity and rapidexcretion into the bile. (See M. L. J. Landsman, G. Kwant, G. A. Mook,W. G. Zijlstra, “Light-Absorbing Properties, Stability and SpectralStabilization of Indocyanine Green”, Jour. Applied Physiology, Vol. 40,No. 4, April 1978). Moreover, ICG is readily soluble in water. Followingintravenous injection, ICG binds to plasma proteins with albumin as itsprinciple carrier. ICG undergoes no significant extra hepatic or enterohepatic circulation. ICG is taken up by plasma almost exclusively by thehepatic parenchymal cells and is rapidly excreted into the bile. Due toits properties, ICG is used extensively for the study of hepaticfunction. ICG is FDA approved and can be used with proper dosages as perFDA regulations. IC-Green™ is available in vials from Akorn Inc. Dosagesfor ICG can also be taken from the same.

It has been demonstrated that ICG can be used as a contrast agent inanimals for visualizing biliary tract during LaparoscopicCholecystectomy. (See Araki, K. Namikawa, J. Mizutani, M. Doiguchi, H.Yamamoto, Arai T. Yamaguchi, et al. “Indocyanine Green Staining forVisualization of Biliary System during Laparoscopic Cholecystectomy”;and, D. Persemlidis, A. Barzilai, et al. “Enhanced LaparoscopicVisualization of Extrahepatic Bile duct with Intravenous ICG”). Thepresent invention uses the fluorescence properties of ICG for viewingthe biliary tract during cholecystectomy.

The spatial resolution of the fluorescence imager was established bycomputing the percent contrast by imaging a standard USAF 1951resolution target. The block diagram for the set up is given in FIG. 5.The distance between the source and the target was set as 22 inches. Theresults are plotted for multiple bins. The percent contrast, C, whereI_(max) is the maximum intensity reflected by a line of the resolutiontarget (i.e., a white bar) and I_(min) is the minimum intensity from thenon reflecting area between the white bars (i.e., dark bars).

Percent contrast is calculated by using equation (1):

$\begin{matrix}{C = \left( \frac{I_{\max} - I_{\min}}{I_{\max} + I_{\min}} \right)} & (1)\end{matrix}$

This percent contrast measurement is carried out with and without theemission filter on the detector end, to observe any effects on spatialresolution due to the filter. FIG. 6A shows the percent contrastmeasurements carried out without the emission filter on the detectorend, while FIG. 6B shows the percent contrast measurements carried outwith the emission filter on the detector end.

It was found that there is a slight reduction in percent contrast whenthe emission filter is placed in the path. This can be attributed to theway the filters are manufactured. The surface of the filters is notalways smooth and hence there can be a reduction in the spatialresolution. With the filter in place, the spatial resolution of thesystem (not for fluorescence) turns out to be 0.4 mm when used with thelowest bin value (Bin 1).

The following describes the quantification of the absorption spectrum ofICG. The first step in determination of ICG as a potential fluorophorefor visualizing biliary tract during Cholecystectomy is to find itsabsorption spectrums and wavelengths needed for ICG to get excited forfluorescence. ICG has a bimodal absorption peak when in an aqueoussolution (e.g., distilled water). The absorption spectrum changes withdifferent solutes of ICG and different concentrations. This observationleads to the conclusion that ICG does not obey Beer Lamberts law. Thereis however a low concentration region where it does obey Beer Lambertslaw. (See M. L. J. Landsman, G. Kwant, G. A. Mook, W. G. Zijlstra,“Light-Absorbing Properties, Stability and Spectral Stabilization ofIndocyanine Green”, Jour. Applied Physiology, Vol. 40, No. 4, April1978). The set up for measuring the absorption spectrum of ICG is shownin FIG. 7.

A QTH lamp, as discussed above, along with all its optical components iscoupled to a low pass filter (e.g., an excitation filter having a cutoffof 790 nm), which acts as an excitation filter for the fluorophore ICG.Cardio Green (Sigma Aldrich, St. Louis, Mo.) was used. Three differentconcentrations of ICG (0.078 mg/ml, 0.015 mg/ml and 0.005 mg/ml) wereprepared. An initial solution of 0.078 mg/ml was prepared and the otherconcentrations were obtained by dilution of this initial solution. Thus,any error in the measurement of ICG is consistent. These were filledinside capillaries having a 1.5 mm bore diameter. The detector (PIXIS400 BR) is coupled to a NIR Liquid Crystal Tunable Filter (LCTF)(Cambridge Research & Instrumentation, Boston, Mass.) which is alreadycalibrated. The distance between the source and the target was set to 22inches for all concentrations. It is possible to tune the LCTF fordifferent wavelengths and get images at each of those wavelengths. Thewavelengths and interval between wavelengths is user defined throughsoftware. In this embodiment, the LCTF was tuned for wavelengths from650 nm to 900 nm with increments of 2 nm. Images were obtained at every2 nm increment of wavelength starting from 650 nm and ending at 900 nm.Thus, a 3-D hyperspectral image cube is obtained with two spatialdimensions and one spectral dimension. These cubes were obtained foreach three different concentrations and also for spectralon (i.e., amaterial having 100% reflectance). Each of these concentration cubeswere then divided by the spectralon cube to obtain absorptions of thedifferent ICG concentrations. The resultant cube was then filtered usinga savitsky-golay filter and the spectrum was plotted by taking a samplearea from the cube that matches the position of the capillary. ICG indistilled water has a bimodal peak absorption spectrum with maximumabsorption occurring at 705 nm and 775 nm, respectively. One importantobservation is the change in the shape of the curves with changes inconcentration. As the concentration of ICG in distilled water increases,the peak at 775 nm tends to become a shoulder and only one peak at 705nm remains. Contrarily, as the concentration decreases the peak at 705nm tends to become a shoulder and only one peak at 775 nm remains. Thiseffect can be attributed to the aggregate formation of ICG when theconcentration is higher. (See M. L. J. Landsman, G. Kwant, G. A. Mook,W. G. Zijlstra, “Light-Absorbing Properties, Stability and SpectralStabilization of Indocyanine Green”, Jour. Applied Physiology, Vol. 40,No. 4, April 1978).

Determination of the fluorescence concentration that produces the bestemission was required. The set up for determination of best fluorescenceconcentration is shown in FIG. 8. A QTH source is coupled to a low passfilter having a cutoff of 790 nm, which provides all excitationwavelengths for ICG. Different concentrations of ICG in distilled waterwere used. An ICG solution having a concentration of 0.03 mg/ml wasfirst made by carefully measuring from an Adventurer™ (Ohaus). Thissolution was further diluted to form the other concentrations (0.02mg/ml, 0.015 mg/ml, 0.010 mg/ml and 0.005 mg/ml). These solutions wereput in different capillaries. A NIR detector (PIXIS 400 BR) coupled toan emission filter having a cutoff of 805 nm was used to measure thefluorescence. The five capillaries were kept parallel to each other andwere excited by the QTH source coupled to the excitation filter asmentioned previously. Care was taken to make sure that the capillarieseach received approximately the same intensity of illumination bykeeping the capillaries closer to each other. Fluorescence photonsemitted by ICG were then detected using the detector mentioned above.The best concentration was decided based on the maximum fluorescencephotons detected by the detector. Higher concentrations were eliminatedas there was a reduction in fluorescence emission and due torestrictions on the dosage.

Fluorescence for 0.03 mg/ml, 0.02 mg/ml and 0.015 mg/ml ICG are almostidentical with the peak fluorescence at 0.03 mg/ml and 0.015 mg/ml.Thus, either concentration can be used for injecting into humans.However, dosage regulations restrict the concentration to 0.015 mg/ml.ICG forms aggregate at higher concentrations. The total dose of dyeinjected should be kept below 2 mg/kg. Thus, for an average adult havinga weight of 70 kg, the concentration of dye with blood as a solvent is0.025 mg/ml.

Maximum depth of fluorescence inside tissue was then determined. Humantissue has both scattering and absorbance properties. Thus, lightfalling on tissue gets partly scattered and partly absorbed. Thisdepends on the absorbing properties of the tissue. Different tissueshave different absorption and scattering properties. As described above,one of the issues related to viewing the bile duct is that the CBD liesunderneath a thick fat layer. Thus, it is important to determine thedepth that the fluorescence imager can see, to enable the user toaccurately determine the position of bile duct.

An intralipid model was then considered, which model mimics tissueproperties. A 1% intralipid solution was prepared by diluting a 20%intralipid solution (obtained from Fresenius-Kabi) with distilled water.This 1% intralipid solution mimics the properties of skin tissue. (See IDriver, J W Feather, P R King, J B Dawson “The optical properties ofaqueous suspensions of Intralipid, a fat emulsion” Phys. Med. Biol.,1989 Vol:34, No 12, 1927-1930). 0.03 mg of ICG was carefully weighedusing an Adventurer™ (Ohaus) and then mixed with 10 ml of distilledwater. This solution was then diluted to get a concentration of 0.015mg/ml. This fresh ICG solution was filled into capillaries having 1.5 mmbore diameters using a 1 cc syringe. The capillary was then fitted to astand, which was coupled to a vernier height gage such that eachcapillary could be moved with precision in the vertical direction. Thevernier height gage enables data to be produced at each millimeter. Thefresh intralipid solution, previously prepared, was then filled into atub. The setup was arranged in such a way that the center point of thebore of each capillary was at the surface of the 1% intralipid solution.A QTH source coupled with the low pass excitation filter having a cutoff of 795 nm was used as an excitation source for the ICG in thecapillary. The detector (PIXIS 400 BR) was coupled to a high passemission filter. The distance between the source target and the detectoris also illustrated in the figure. A distance of 22 inches between thesource and target was fixed because of clinical constraints duringcholecystectomy surgery. The detector was focused on the surface of theintralipid solution. At each step, each capillary was carefully movedinto the intralipid solution in increments of 1 mm using the vernierscale. Care was taken not to disturb the alignment of the capillary tubewith respect to the detector. Two different studies were done using thesame setup: (1) with a constant exposure time; and, (2) with variableexposure time. In the constant exposure method, the exposure time andaperture of the detector was adjusted to measure maximum fluorescencewithout getting saturated, when each ICG filled capillary was at thesurface of intralipid solution. The same exposure time and aperture wereused for all depths. Background (i.e., plain intralipid solution usingboth filters) images were also collected using the same exposure andaperture. In the variable exposure time method, the exposure time wasvaried at each depth to get maximum fluorescence photons out of the ICG.Thus, each depth had a different exposure time. This difference inexposure time was accounted for during the analysis of the images.Background (i.e., plain intralipid solution using both filters) imageswere also collected using the same aperture but different exposure timesto obtain maximum counts from the intralipid solution. The entire studywas conducted in the dark to maximally avoid any stray light from thebackground into the detector.

Data acquisition was also taken from beef fat. Data was taken withcapillaries having aqueous ICG, at 2 mm from the surface of the fat andat 2 cm from the surface of the fat. The second data point can be usedas a threshold to determine the penetration depth in the intralipidmodel as shown in FIGS. 9A and 9B.

Images at different depths were collected in both methods. Imagescollected using method 1 (i.e., with constant exposure), were ratioedwith the background image obtained with all parameters being the same.However, in the case of method 2 (i.e., variable exposure time), eachimage was first divided by its exposure time to obtain an Intensity/secat each pixel. The background image was also divided by thecorresponding exposure time and then each image was ratioed with theprocessed background image. Ratioing with the background image removesany source patterns associated with the original images. Thus, thefluorescence signals can then be processed. Beef fat was obtained and acapillary filled with ICG having the optimal concentration obtainedabove at approximately 2 mm to 3 mm below the surface of fat. The imagesobtained are shown in FIGS. 9A and 9B.

FIGS. 9A and 9B show images of beef fat with an ICG in water filledcapillary taken from two cameras. FIG. 9A shows a digital image of beeffat with a capillary having ICG with water in it. FIG. 9B shows adigital image taken using the Surgical Fluorescence Imager of the samebeef fat with the same capillary. The fluorescence glow can be clearlyseen.

In both methods, the processed images were then cropped for the regionof interest, and then a row profile of each row was obtained. Thisprofile was median filtered with a filter window of 3. The signal tonoise and contrast to background at each row were then averaged andplotted along with their error bars. FIG. 10 shows SNR and contrast tobackground calculations. This profile is taken from each row of theimage. SNR and contrast to background are then calculated for each rowand then averaged.

SNR (Signal to Noise Ratio) is the ratio of signal power to the noisepower. From the profile obtained above, signal to noise was calculatedby selecting the noise region, getting the standard deviation and thendividing the signal by this noise. This is depicted in FIG. 10, anddescribed by equation (2):

$\begin{matrix}{{SNR} = \left( \frac{{Sig}_{\max} - {Sig}_{\min}}{{StdDev\_ of}{\_ Noise}{\_ Region}} \right)} & (2)\end{matrix}$

where

-   -   Sig_(max) is the maximum fluorescence signal; and,    -   Sig_(min) is the baseline fluorescence signal.

The plots for the SNRs using both of the above mentioned methods areshown in FIG. 11A and FIG. 11B. SNR was calculated for each row from thecropped, processed image. A Mean SNR was calculated along with thestandard deviation. FIG. 11A shows the signal to noise ratio againstpenetration depth in the intralipid solution of aqueous ICG withconstant exposure, while FIG. 11B shows the signal to noise ratioagainst penetration depth in the intralipid solution of aqueous ICG withvariable exposure. The square data points depict aqueous ICG in beef fatat two different depths.

Moreover, FIGS. 11A and 11B show that the signal to noise decreases asthe capillary goes deeper. As discussed supra, intralipid has bothabsorption and scattering properties. When the fluorescence signal goesdeeper, scattering and absorption takes place in the intralipid leadingto lower signal and higher noise. Another factor that could affect thesignal is the excitation source. The penetration depth of the excitationlight itself is lower due to intralipid absorption and scatteringthereby leading to lower fluorescence signal and a reduction in SNR.Typically the threshold for SNR is taken as five (5). As shown in thefigures, a SNR of five (5) is reached at 18 mm below the surface withconstant exposure and at 21 mm below the surface with variable exposure.

Another measure of depth analysis, with the same set up was taken forICG with bile. Care was taken to maintain the concentration of ICG inbile to be 0.015 mg/ml. This was carried out with the constant exposuremethod. FIG. 12 is a plot depicting the signal to noise ratio againstpenetration depth in intralipid of ICG with Human Bile using theconstant exposure method.

When bile is mixed with ICG, the penetration depth reduces due tovarious effects of the bile, e.g., quenching. Thus, from the plot, wecan see that an SNR of five (5) is reached at 11 mm below the surface.Beyond this depth, it is difficult to distinguish between signal andnoise, and thus data analysis was terminated.

Contrast to Noise was calculated using the same value of signal as forSNR and dividing it by the mean of the noise region. This parameter canalso be used as a deciding factor for finding the penetration depth ofICG in intralipid. The plots for contrast to background are shown inFIGS. 13A and 13B. FIG. 13A shows a plot depicting the contrast tobackground ratio against penetration depth in intralipid of aqueous ICGusing the constant exposure method, while FIG. 13B shows a plotdepicting the contrast to background ratio against penetration depth inintralipid of aqueous ICG using the variable exposure method. Squaredata points show data with aqueous ICG in beef fat at two differentdepths.

Similar to signal to noise ratio, contrast to background ratio was alsocomputed for ICG with Bile in an Intralipid solution. FIG. 14 is a plotdepicting the contrast to background ratio against penetration depth inintralipid of ICG with Human Bile using the constant exposure method.

From FIG. 14, it is clear that the contrast to background ratiodecreases with penetration depth. From FIGS. 11A-14, it can be concludedthat the penetration depth of ICG with water in a 1% intralipid solutionis approximately 21 mm from the surface of intralipid. The dashed linein the plot (FIG. 13B) determines the threshold value of contrast tobackground taken from aqueous ICG in beef fat. However, since a 1%intralipid solution is a homogenous medium and ICG in aqueous solutionis also homogenous, the penetration depth is high. Analysis of beef fatrevealed a contrast to noise threshold of 1.0.

From FIG. 14 it is clear that the penetration depth is approximately 11mm below the surface of intralipid beyond which the contrast tobackground ratio is below 1.

As described above, free bilirubin does not have high fluorescence andconjugated bilirubin has a better quantum yield. Bilirubin has a strongand main absorption spectrum in the visible region with wavelengthsranging from 390 nm to 460 nm and a molar absorption coefficient ofapproximately 50×10³ mol⁻¹ cm⁻¹ in aqueous solvents. The position ofmaximum absorption, the shape of the spectrum and the molar absorptioncoefficient depends greatly on the conformational structure.

The fluorescence of bilirubin increases in the presence of albumin. (SeeHumra Athar, Nisar Ahmad, Saad Tayyab, Mohammad A. Qasim “Use ofFluorescence enhancement technique to study bilirubin-albumininteraction” Int. J. Biological Macromolecules, 25, 353-358, 1999).Bilirubin dianions combine reversibly with human albumin in neutral oralkaline solution. (See Jorgen Jacobsen, Rolf Broderson“Albumin-Bilirubin Binding Mechanism” Jour. Of Biological Chemistry, Vol258, No. 10, Issue of May 25, pp. 6319-6326, 1983). A bilirubin albuminsolution has an absorption spectrum from 500 nm-600 nm, with an emissionoccurring at around 528 nm.

Measurement of fluorescence was done using the source detectorcombination as shown in FIG. 15. FIG. 15 is a block diagram depicting aset up for measurement of fluorescence from bilirubin.

Bile was taken from a patient with the patient's consent. The raw bilewas used for filling the capillary through a syringe. The capillary waskept over a black cloth to avoid any reflection as this would affect thefluorescence. A source coupled to an excitation filter (i.e., a shortpass filter having a cut off of 500 nm) was used to excite the bileinside the capillary. The source was a broad band QTH source asdiscussed above. The source-filter combination was at a distance of 15inches from the capillary. The detector used was CoolSnap_(ES)(Princeton Instruments). The detector was coupled to the bilirubinemission filter (i.e., a long pass filter having a cut off of 515 nm)along with a 50 mm, f/1.4 Nikon lens. The detector and the source werearranged in reflective mode geometry as shown in FIG. 15. The detectorwas further connected to the laptop for data acquisition. In such amode, light emitted through fluorescence was collected at the detectorwhich then amplified and provided an image. FIG. 16 is a fluorescenceimage of bilirubin with excitation from 400 nm-500 nm. The fluorescenceis clearly seen as compared to the blank capillary.

The above described image was obtained by subtracting the raw imageagainst its background (i.e., image without the bile capillary). Fromthe image, the fluorescence of bilirubin is clearly seen. This can beused as one of the parameters in the detection of biliary duct. Fromthis experiment, it has been found that the bilirubin in the bilefluoresces. This provides another method of visualizing the bile duct ina cholecystectomy procedure.

The spatial resolution of the bilirubin fluorescence imager wasestablished in a similar manner as it was done for determining percentcontrast using the ICG fluorescence imager. However, in this embodiment,the detectot was the CoolSNAP_(ES). Excitation filters having a cut offof 500 nm and emission filters having a cut off of 515 nm were used. Theresults are shown in FIGS. 17A and 17B. FIGS. 17A and 17B are plotsdepicting percent contrast against spatial resolution, when no emissionfilter is used (FIG. 17A), and when an emission filter is used (FIG.17B). FIGS. 17A and 17B show that there is a slight decrease in thepercent contrast when the emission filter is used in the optical path.This is possible as, there can be non-uniformity generated due to thefilter from certain manufacturing issues. Additionally, the spatialresolution with the emission filter is approximately 0.25 mm (at apercent contrast of 26%). This does not reflect the spatial resolutionwhen measuring fluorescence and just reflects the spatial resolution ofthe CCD coupled to the filter.

It has been demonstrated that Indocyanine Green and Bilirubin can beused as a latent contrast agent for visualizing an anteriorly placedbiliary structure. This is evident from the analysis of penetrationdepth in an intralipid model. However, ICG mixed with bile resemblescloser to real data imaging. Thus, it has been shown that thepenetration depth of Indocyanine Green is approximately 11 mm below thesurface of a 1% intralipid. Plain aqueous ICG was used for depthanalysis in a 1% intralipid solution, and the penetration depth isalmost doubled. This demonstrates that when bile is mixed with ICG,aggregates are formed. Moreover, bile itself absorbs some of theincident radiation that falls on it, thereby reducing the excitationenergy for ICG. However, this forms a more realistic model that can beused for clinical studies.

Another potential method is to perform multimodal imaging using a DLPhyperspectral imager that is capable of imaging fluorescence signalsfrom ICG, bile and NIR reflectance radiation. Moreover, the presentinvention comprises a method of performing multimodal imaging using aDLP hyperspectral imager which includes but is not limited tovisualization of biomarkers and/or biochemical markers that absorb,reflect and/or fluoresce by using digital signal processing andchemometric algorithms, or alternatively, complex spectral illuminationcan be used to collect complex spectral data thereby providingchemically encoded information. Accordingly, an embodiment of thepresent invention includes a method of confirming the location of thebile ducts by video rate or near video rate imaging of the area andswitching between various methods such as the hyperspectral imagingmethod using NIR spectral illumination and fluorescing ICG and bile.

FIG. 18 is yet another embodiment of the hyperspectral imager of thepresent invention. Briefly, the hyperspectral imager includes Xenon lampassembly (Optronic Labs) 200, broadband light 202 emitted from lamp 200,passes through sub-millimeter slit (Optronic Labs) 204 and in turnpasses through collimating lens 206. Collimated light 208 then reflectsoff of diffraction grating 209 which creates diffracted light 210, i.e.,light separated into a plurality of bandpasses of light. It should beappreciated that slit 204, in combination with collimating lens 206 andgrating 209 may also be a prism, a tunable filters, an electromechanicaloptical filter wheel, an acousto-optical tunable filter, aliquid-crystal tunable filter, a digital micromirror device, anelectro-optical filter, a holographic filter, and combinations thereof.The diffracted light strikes a DLP® digital micromirror array 211, whichreflects projected light 212. Projected light 212 includes intensitiesof wavelengths of a complex spectrum reflected back into the opticalpath by array 211. In other words, diffracted light 210 falls on array211 in such a way that the columnar bands of discrete wavelengths oflight each fall on a specific column of micromirror on array 211. Then,by selecting particular rows within the columns of micromirrors, thespectral content of projected light 212 may be controlled forming acomplex spectral illumination. For example, as shown in FIG. 18, rowswithin the columns of micromirrors corresponding to wavelengths rangingfrom 380 nm to 1600 nm are “turned on” while rows within other columnsof micromirrors are “turned off”. Thus, projected light 212 includesvarying light intensity ranging in wavelength from 550 nm to 600 nm.Subsequently, projected light 212 passes through beam shaping optics 214and strikes tissue sample 216. Reflected light 218 from tissue sample216 is received by CCD Focal Plane Array detector 220 and the detecteddata is communicated to computer 222 for data processing. Computer 222can also be arranged to control digital micromirror array 210 and thedetector 220.

As described supra, the present invention includes a spectralilluminator using DLP technology to excite fluorescence markers (whichcan be native to the tissue (tissue fluorescence) or target, or that wasinjected or painted onto the tissue or target) and an optical filter,LCTF or other element used to pass emitted fluorescence light to adetector. With the improved imaging provided by the present invention,surgical procedures can focus on the target for treatment (or itsadjacent tissue) by visualization, e.g., the biliary tree. The presentinvention system can be used for other surgical applications to performin vivo pathology during surgery, or alternatively, in vivohyperspectral pathology. The present invention can also be used inmethods, including, in vivo pathology of cancer in real-time (e.g.,before, during or after surgery), in the body (e.g., laparoscopically)without having to biopsy and/or fix the tissue or target to a slide.Other examples include a probe that binds to antigens. The techniquesare also applicable to fluorescence microscopy. Finally, chemometricsand digital signal processing may be used to enhance the visualization.Furthermore, it has been found that using a digital light processor thepresent inventors were able to achieve video rate hyperspectral imagecapture and processing. Video rate capture and processing is notpossible with current illumination technology and made possible byincluding DLP. The video capture rate also benefits from the chemometricillumination.

Clinical results have shown visible reflectance hyperspectral imaging iscapable of visualizing chemical changes using inherent chromophoreswithin the microvasculature. These results have created the opportunityto apply this cutting-edge imaging modality in a variety of clinicalapplications. As an example, the surgeon needs a system for helpingdetermine the degree of amputation and monitoring vascular healing aftersurgery in order to reduce the risk of an amputation failure. Similarly,plastic and reconstructive surgeons have a need for a reliable systemindicating tissue viability of skin flaps and transplanted tissue, aswell as, determining how much tissue to extract when removing acancerous skin tumor. Moreover, the laparoscopic gastric bypass surgeonneeds a system for detecting hemorrhage and ischemic tissue duringgastric bypass procedures to avoid catastrophic complications. To thatend, hyperspectral imaging is useful in monitoring and assessingvascular healing after performing a lower limb amputation, indetermining the degree of amputation and tumor removal, monitoringtissue viability of skin flaps and transplanted tissue and detectingstrictures and leaks during gastric bypass surgery. There wereapproximately 82,000 lower-limb amputations reported in the year of2002; 105,000 bariatric gastric bypasses; 500,000 cholecystectomysurgeries were performed in 2003; and, approximately 59,350 new cases ofskin cancer, all of which are expected to increase over time.

As described supra, hyperspectral imaging is an optical imagingtechnique that captures the spatial and spectral information from thesource target, typically capturing hundreds, of contiguous wavelengthbands for each pixel. The hyperspectral data would be rendered useless,until the recorded data is precisely analyzed and processed using toolsborrowed from spatial image processing, chemometrics and spectroscopy toyield information that can be presented in an image form. Thus, thehyperspectral imaging approach provides a “data cube” consisting of twodimensional images with each desired wavelength being represented by oneof the images from the stack of images collected. The image cube thusobtained, has three dimensions: two (2) spatial (X and Y) and one (1)spectral dimension (wavelength). A basic hyperspectral imaging systemconsists of a broadband light source, with the desired spectral band ofinterest; optics arranged to focus light on the source target; anelectronically tunable filter to spectrally discriminate the imagedlight from the source target; and, a sensitive array detector or focalplane array, which collects the light, converts it into a twodimensional gray scaled image and transfers it to a computer. The imageacquisition process continues until all images at desired wavelengthshave been collected, generating the hyperspectral image cube. Anoverview of a hyperspectral image cube is depicted in FIG. 19.

Hyperspectral imaging has been used in satellite technology to identifyfeatures on the earth surface; however, there has been littleapplication to biology and almost none to clinical medicine. The presentinventors address this with the next generation of hyperspectral imagingsystems that are “source based”.

The present invention incorporates DLP® technology and increases thespeed of image data acquisition. The DLP® is used in conjunction with asource lamp, and is capable of tuning different wavelengths of lightwithin microseconds versus milliseconds required by current systems inthe industry. In addition, the placement of the liquid crystal tunablefilter (LCTF) within the slit lamp beam path and the spectraldeconvolution are unique features of this invention.

The current invention source based system can collect data 60 to 100times faster that previous known detector based systems. With theincreased speed, the present invention may be applied to practicalclinical and surgical uses. Additionally, the increased speed reducesmovement artifacts, producing real time video images, with thecapability of displaying both normal and hyperspectral videosimultaneously.

The present invention includes integrating DLP® technology with FocalPlane array technology, collecting and storing digital hyperspectralimage data and analyzing the data using chemometric analysis methods.Visualizing tissue chemistry are features unique in the field ofclinical hyperspectral imaging. This technology enables clinicians tovisualize the levels of chemicals within the tissue, for example,visualizing anteriorly placed or hidden structures during open,endoscopic and laparoscopic surgery detecting retinal disease early,guiding laser therapy, monitoring patients during pharmacologictherapies for preventing blindness, helping guide and monitor woundhealing and amputations as well as skin flaps during recovery. Otherapplications include research grade devices for animal research and inquality control of pharmaceuticals and food.

Another example application of the present invention is to visualizetissues and organs in the human body using chemometric techniques andsophisticated signal processing methods. The system has a wide range ofclinical applications such as enabling surgeons to see through tissuesbefore they cut into them during cholecystectomies, chromoendoscopyfacilitating cancer detection, plastic surgical skin flap viability andburn wound healing evaluation, skin cancer detection, retinal bloodperfusion in diabetic retinopathy, assessing and monitoring woundhealing in lower limb amputations, monitoring anastomotic viability ingastric bypass operation, dental evaluation, measuring oxygenation ofthe kidneys and monitoring pressure sores. The present invention canalso be used as an integral component to the smart bed and smarthospital for measurement of vital signs and monitoring a patient'stissue oxygenation.

The present invention consists primarily of a spectral light engine withDLP® technology providing the spectroscopic illumination, a digitalcamera with a scientific grade CCD for imaging, and software designedand developed to manage the data acquisition and the chemometricvisualization. The data acquisition software automatically tune the DLP®technology and trigger the camera for collecting a series ofspectroscopic images formatted as a hyperspectral image cube. Next, thespectroscopic image data are deconvoluted using chemometric analysismethods. The resulting gray scale or color encoded images provides theclinician with a non-invasive visualization of the chemical state withinthe micro vasculature perfusing the tissue while the patient is in theclinic or surgery.

One of the many goals of the present invention is to demonstrate aDLP®-based Visible Hyperspectral Imaging Systems can be used routinelyfor a variety of medical applications. In certain embodiments, a DLPspectral light engine can be used for illuminating the tissue withdifferent wavelengths, colors of light and light reflected back to thedetector are measured as a spectrum at each image pixel and usingdifferent chemometric methods to determine relative levels of inherentchromophors within the tissue can be visualized. In addition, thechemometric methods can be performed on the detector instead of on thesource, or the combination of both. Furthermore, the present inventioncan illuminate the object to be analyzed with spectroscopic light, tunedby DLP®. The speed of the present invention reduces the currentacquisition time from minutes to seconds making hyperspectral imaging apractical everyday surgical and clinical tool for imaging.

In some embodiments, the present invention may be used in the field ofopthalmology. Here, a hyperspectral slit lamp, which involves theintegration of electro-optics and a clinical slit lamp, enablesclinicians to image psychopathological aspects of the retina, includingthe oxygenation status and the level of macular pigments. This in turnallows the clinician to assess the need for treatments earlier than iscurrently possible. Additional examples of the clinical applications ofhyperspectral imaging include: (a) the determination of areas ofischemia in diabetic retinopathy and retinal vascular occlusivediseases. This helps the clinician to decide if laser therapy orintraocular injections of pharmacologic agents would be helpful toprevent or treat retinal neovascularization or macular edema. In thecase of laser therapy, this imaging technique would also help delineatethe specific areas of the retina that require treatment; (b) themeasurement of macular pigment levels to help in the early diagnosis andtherapeutic interventions of diseases like parafoveal telangiectasisand, most importantly, age-related macular degeneration; and, (c)monitor patients for determining the best pharmacological therapy formaximizing favorable patient outcomes. Consequently, hyperspectralimaging may help prevent or decrease the vision loss from these highlyprevalent diseases.

In some embodiments, the present invention capitalizes on differentlevels of chemicals, for example the relative amount of oxyhemoglobin,deoxyhemoglobin, carboxyhemoglobin, and met perfusing the retinaltissue. Imaging not only the vasculature structure but the relativeamount of chemicals within such structure to help detect and diagnoseeye disease early. The present invention can directly imaging the levelof the above mentioned molecules particularly oxygenation of the retina.This enables monitoring effectiveness of a therapy of a patient duringtreatment.

This embodiment enables clinicians to visualize the levels of chemicals(oxyhemoglobin, deoxyhemoglobin and carboxyhemoglobin within the retina,to detect retinal disease early, to guide laser therapy and to monitorpatients during pharmacologic therapies for preventing blindness. Inthis embodiment, a liquid crystal tunable filter is placed into the beampath of a slit lamp source illuminating the eye with differentwavelengths of light and light reflected back to the detector aremeasured as a spectrum at each image pixel. This is a uniqueconfiguration designed specifically for measuring the chemical levelswithin the retina.

Typically, the wavelengths to be used depend on the tissue type due tothe penetration of tissue depth. For example, in visible spectra, HbO₂,Hb and HbCO₂ can be measured. In near-infrared (NIR) spectra, HbO₂, Hb,water and lipids can be measured.

In another embodiment, the present invention can be used for detecting,diagnosing and monitoring disease in live human patients visiting theclinic or during open and closed (endoscopic and laparoscopic) surgicalprocedures. The system can also be used for animal researchapplications. A hyperspectral imaging system integrating DLP technologyto illuminate the area of interest with wavelength bands of lightranging over a spectral range from the visible to the near-infrared thatis interfaced with a digital focal plane array for acquisitioning andstoring a series of digital spectroscopic images (a hyperspectral datacube) that is analyzed using chemometric methods providing chemicalinformation within the area of interest aiding the clinician indetecting, diagnosing and monitoring disease. Clinicians can alsomonitor patients with this technology for determining the bestpharmacological therapy and maximize favorable patient outcomes. Thepresent invention utilizes DLP technology with a focal plane array andchemometric deconvolution for visualizing the biochemical levels withina human non-invasively for clinical and surgical applications. Thepresent invention can also be illuminated with spectrum of light.

In yet another embodiment, the present invention enables the clinicianto visualize anatomical structure and the chemistry within live humanpatients or research animals within seconds during a visit to the doctoror during open, laparoscopic and endoscopic surgery. These images areuseful to the clinician or surgeon for detecting, diagnosing, andmonitoring disease and the effectiveness for identifying hiddenstructures during surgery and help clinicians determine the besttherapy. The present invention provides unique capability to imagerelative levels of inherent chromophors within the tissue noninvasivelywithout contrast agents in live humans during clinical visits or duringsurgery. A DLP spectral light engine may be used for illuminating thetissue with different wavelengths spectrum of light, and the lightreflected back to the detector are measured as spectrum at each imagepixel and using different chemometric methods to determine relativelevels of inherent chromophors within the tissue can be visualized.

In certain embodiments, the system of the present invention has multiplemodalities that include UV light, visible light, near infrared light,infrared light, and fluorescence excitation. In the DLP®-based VisibleHyperspectral Imager (DVHI) embodiment, the present invention may becoupled with standard surgical and clinical devices, for example alaparoscope, slit lamp, or with lens for microscopic imaging. Inparticular, a quick release mechanical coupling between the DVHI and astandard laparoscope is developed allowing the system to image bothlaparoscopically during closed surgical procedures and removing thelaparoscope for open surgery imaging. The DVHI system is calibrated atNIST and to NIST standards. For visible to 2500 nm applications, thepresent invention may use TI Discovery 1100 electronics board with anAccessory Light Processor (ALP) electronic board.

In another embodiment, the DVHI is used in vivo using an animal modelduring open surgical procedures and during closed laparoscopicprocedures. Spectral light illuminating the animal tissue are evaluatedfor safety by measuring the temperature of the tissue. A variety oftissues can be imaged and identified by measuring spectroscopy and thechemometrics, as well as, imaging before and after inducing ischemia andhemorrhaging the tissue. The system can be assessed by comparinghyperspectral and chemometric images with an evaluation made by theattending veterinarian using standard clinical methods.

Yet in another demonstration, the DVHI is translated to human surgeryand clinic establishing its efficacy for a variety of humanapplications. Here, the present inventors translate Visible DLP®Hyperspectral Imaging for imaging during live human surgery, andmonitoring the progression of clinical disease for determiningfeasibility. Also, surgical and clinical hyperspectral data is monitoredto observe the progression of disease and changes in tissue chemistryduring wound healing due to trauma, amputations, tumor removals, plasticsurgery skin flaps and visualizing ischemia and hemorrhage duringgastric bypass. For these purposes, the present inventors utilized theexisting chemometric analysis algorithms and develop new visualizationmethods, chemometric and statistical analysis. In addition, the presentinventors continue collected clinical hyperspectral image data usingexisting system and existing hospital protocols for building ourclinical hyperspectral image database that can be populated over timespecifically on spectral properties of composite structures for examplethe gallbladder and extrahepatic bile ducts, determined additionalclinical applications, and refined new chemometric algorithms. The endresults provide better imaging algorithms for better clinicalvisualizations that can be implemented into the software of the new DLP®Visible Hyperspectral Imaging System.

The DLP® Hyperspectral Imaging and Chemometric deconvolution can beapplied to all products that use light, for example, the fields ofclinical endoscopy, clinical chemistry, microscopy, surgical microscopy,drug discovery, microarray scanners and microplate readers.

In certain embodiments, the multi-modal microscopic reflectanceHyperspectral Imaging system developed provides enhanced diagnosis in aclinical environment. The system developed has the capability to measurerelative contributions of oxyhemoglobin in the microvasculature in thevisible as well as the near infrared region, hence it is multi-modal. Anexample was formulated to measure contributions of oxyhemoglobin in themicrovasculature perfusing the dermal tissue of the palm in the nearinfrared region. The system was also used to image the human eyemeasuring the relative contributions of oxyhemoglobin perfusing thescleral surface. The instrumentation used for this imaging system isshown in the FIG. 20.

FIG. 20 depicts one embodiment of the above mentioned hyperspectralimaging system of the present invention having charged coupled device300 (CCD), beam splitter 302, eye pieces 304, magnification knob 306,light source 308, joystick 310, intensity control knob 312, illuminationmirror 314, subject position 316, liquid crystal tunable filter (LCTF)318, and common center for rotation 320. In another embodiment, the LCTFmay be replaced with a digital micromirror device, and can be locatedanywhere along the optical path of the present apparatus.

In certain embodiments, the instrumentation shown in FIG. 20 may containa slit lamp (light source 308) that illuminates the target using anin-built 12V/30W illumination Halogen lamp source. The light from thesource is focused onto the target which is placed on the Y-shapedheadrest. The diffused reflected light from the target is passed backthrough the slit lamp microscope optics onto a beam splitter 302 whichallows a part of light into eyepiece 304 for the observer and theremaining part of light to the relay optics which further pass lightinto the LCTF 318, which is an electronically tunable filterdiscriminating the reflected light into individual wavelengths passingit onto the camera lens which then focuses it onto Charged CoupledDevice or HQ2 (CCD) camera 300. It should be appreciated that theforegoing arrangement, which includes a LCTF, may alternatively includea DLP illumination device as described throughout this application.Thus, the LCTF can be removed, and the DLP illumination device coupledto this system at the location shown for light source 308. The imagesformed on the CCD are then digitized and stored in a laptop computer forfurther analysis. In some embodiments, the system can be applied tocollect data in the near infrared region or in the visible region. Thevisible and near infrared application differ in the use of a unique CCDcamera, i.e., the CooLSNAP_(ES) camera instead of the PIXIS camera, anda unique LCTF, visible LCTF instead of near infrared LCTF, or visibleversus a NIR DLP® spectral illuminator.

The slit lamp is a microscope having an illumination system and anobservation system, which are mounted on a rotated drum with a commoncentre of rotation. The slit lamp is binocular; that is, it has twoeyepieces 304, giving the binocular observer a stereoscopic (i.e., threedimensional) view of the eye. These devices are sometimes referred to asBiomicroscope or stereomicroscopes. The Galilean magnification changerallows increasing the magnifying power between three stages, e.g., 10×,16×, 25×, by simply turning a knob which is designed with a flat area sothat the operator can change the power by feel, without looking at themarkings, since the higher magnification is achieved when the flatportion is on top, while lowers magnification is achieved when the flatportion is at the bottom. The actual magnification of what is seenthrough the slit lamp is derived by multiplying the power of theeyepieces, which are fixed, with the power of the objective lens.

Thus, if the eyepieces 304 are 10× and the objective lens is 1.6×, thetotal magnification is 16×. The non-radiometric light source of the slitlamp is a 12V/30W halogen lamp providing broadband white light. Thelight is controlled by a transformer which provides continuous lightintensity with the help of rheostat placed next to the joystick 310. Theslit width and the slit height can be continuously varied from 1-14 mm.A Beam splitter 302 is effectively a semi-silvered mirror which isplaced in-between the eyepiece and the microscope optics to partiallytransmit light to the eyepiece and partially reflect light to theattached CCD camera 300. In another embodiment, the detector can be afocal plane array. Typically, for macroscopic and endoscopicembodiments, the present invention uses a non-radiometric source system.This non-radiometric source provides significant advantages for the usein standard clinical operations due to its safety characteristics.However, in the embodiments of the hyperspectral imaging system whichinclude DLP®, a radiometric power supply is used due to it stablecurrent as a light source.

Attaching LCTF 318 and CCD camera 300 onto beam splitter 302 of the slitlamp does not use a lot of CCD chip area, i.e., the spot created by thereflected light from the slit lamp falling onto the CCD after passingthrough the LCTF is very small. Hence there is a need to magnify thereflected light coming out of the beam splitter before passing onto theLCTF. Thus, a relay optic module was designed which is composed of twooptics namely an eyepiece and a photo eyepiece placed in a machinedmetal cylinder. An eyepiece lens is an optical system which is used tomagnify images in optical instruments such as telescopes, microscopes,etc. A photo eyepiece performs the function of focusing the images ontoa plane. These two optical components are housed in a metal cylinder 18cm in length.

Light passes through a Liquid Crystal Tunable Filter (LCTF) 318 havingsolid-state construction and no-moving parts or vibrations, which worksby applying voltage to its liquid crystal elements, to select atransmitted wavelength range, while blocking the rest of thewavelengths. The 60 mm, f/2.8 D micro-Nikon camera lens focuses thelight filtered by LCTF 318 onto CCD camera detector 300. Typically, the60 mm is the dimension for a slit-lamp in spectral detection. The 50 mmlens is used for macroscopic imaging spectral detector, and no lens isused with the LCTF as the spectrum illuminator. It has the ability toproduce distortion-free images with superb resolution, sharpness andcontrast. The lens has a good depth of field and a small workingdistance.

The charge coupled devices (CCD's) or focal plane arrays (FPA) performthree essential functions: photons are transduced to electrons,integrated and stored, and finally read out. They can be roughly thoughtof a two-dimensional grid of individual photodiodes (called pixels),with each photodiode connected to its own charge storage “well.” Eachpixel senses the intensity of light falling on its collection area, andstores a proportional amount of charge in its associated “well.” Oncecharge accumulates for the specified exposure time, the time betweenstart acquisition and stop acquisition (also known as integration time),the pixels are read out serially.

As described in part above, the CoolSNAP_(ES) monochrome camera typicalincorporates a SONY ICX-285 silicon chip with Interline-transfercapability. The interline-transfer CCD has a parallel registersubdivided into alternate columns of sensor and storage areas. The imageaccumulates in the exposed area of the parallel register and during CCDreadout the entire image is shifted under the interline mask into ahidden shift register. Readout then proceeds in normal CCD fashion.Since the signal is transferred in microseconds, smearing isundetectable for typical exposures. However, a drawback tointerline-transfer CCDs has been their relatively poor sensitivity tophotons since a large portion of each pixel is covered by the opaquemask. As a way to increase a detector's fill factor, high-qualityinterline-transfer devices have microlenses that direct the light from alarger area down to the photodiode. Blooming is the migration ofelectronic charge to the adjacent pixels; however, the incorporated Sonychip by Roper Scientific provides protection against blooming by havingbuilt in drains that remove any excessive charge generated from anoverexposed pixel. This Sony interline chip provides anti-blooming foroptical signals greater than 1000 times the full well capacity. Thespectral response for the CoolSNAP_(ES) camera within the visible regionis of interest for the visible system application. The quantumefficiency is always beyond 45% in this region, peaking to over 60%between wavelength regions of 475 nm to 625 nm.

The CCD cameras, PIXIS 400 BR (or 1024 BR), CoolSNAP_(ES) or HQ2 aredriven by PVCAM software which stands for Photometrics Virtual CameraAccess Method. HQ2 has a 20 MHz analogue to digital digitizer which isfaster. This software is used to control the cameras and acquire datafrom them. The data collection process is automated using Vpascalprogramming language integrated in V++, a precision digital imageprocessing and enhancement software. V++ has been designed to controlany PVCAM-compatible feature using the V++ interface and the VPascalbuilt-in programming language. The HQ2 system typically has a 20 MHx A/Ddigitizer that is faster than the others. Another computer codec hasbeen written to configure a portable computer to provide mobile datacollection. The software further controls and synchronizes the speed ofillumination, data collection, analysis and/or deconvolution.

In an embodiment, the software performs the following steps for analysisand deconvolution. First, ratio data for each pixel is calculated.

$\begin{matrix}{{RD}_{ij} = {{Log}_{10}\left( \frac{{BKG}_{ij} - {DF}_{ij}}{{SD}_{ij} - {DF}_{ij}} \right)}} & (3)\end{matrix}$

where

-   -   RD_(ij) is the ratio data for each pixel i at wavelength j    -   BKG_(ij) is the reflectance of a 100% reflectance standard    -   DF_(ij) is a dark field (the value being read when no light is        coming into the camera)    -   SD_(ij) is the reflectance from the sample        The ratioed spectra are then filtered using a Savitsky-Golay        filter, which filter is well known in the art and therefore is        not discussed in detail herein. Next, the spectrum at each pixel        is normalized.

$\begin{matrix}{{ND}_{ij} = \left( \frac{{RDF}_{ij} - {\min\left( {RDF}_{ij} \right)}}{{\max\left( {RDF}_{ij} \right)} - {\min\left( {RDF}_{ij} \right)}} \right)} & (4)\end{matrix}$

where

-   -   ND_(ij) is the normalized spectrum at each pixel    -   RDF_(ij) is the ratioed data        The effect of such normalization is that each spectrum will then        range from 0 to 1. Then, the data is further modified by a        multivariate least squares deconvolution. It should be        appreciated that such a deconvolution requires two or more        reference spectra in order to be performed. In short, a least        squares fit is performed, at each pixel, of the measured        spectrum by performing linear combinations of the reference        spectra to obtain a best fit curve. Then, the resulting relative        linear contributions are scaled to produce a gray scale image.        The following equations are used to performed the foregoing        multivariate least squares deconvolution.        S _(iλ)=(A _(λi) ^(T) ·A _(λi))⁻¹ ·A _(λi) ^(T)  (5)

where

-   -   S is the pixel sensitivity matrix    -   A represents the matrix of i pure component spectra composed of        λ wavelengths        Then, substitution of the sensitivity matrix, S, into the        multivariate regression determines C, comprising linear        contributions of the pure components in the unknown sample,        namely:        C _(ij) =S _(iλ) R _(jλ) +e _(ij)  (6)

where

-   -   R is the matrix of the experimentally determined sample spectra        j    -   e_(ij) represents the residuals.

In another embodiment, the software performs the following steps fordetermining the illumination spectra and subsequent analysis. First, atleast two reference spectra, as measured at the detector, are obtainedfor targets of interest, e.g., 100% oxyhemoglobin and 100%deoxyhemoglobin. It should be appreciated that the targets of interestmay include but are not limited to reference spectra, a patient, aportion of a patient, a purified substance, etc. Additionally, areference spectrum, as measured at the detector, is obtained for 100%illumination from the illumination source. In other words, a measurementof the spectral content of the illumination source is determined. Then,each spectrum is ratioed according to the following equations:

$\begin{matrix}{{{ratioed\_}{spectrum}}_{1} = \left( \frac{{spectrum}_{1} - {spectrum}_{2}}{{spectrum}_{100}} \right)} & (7) \\{{{ratioed\_}{spectrum}}_{2} = \left( \frac{{spectrum}_{2} - {spectrum}_{1}}{{spectrum}_{100}} \right)} & (8)\end{matrix}$

where

-   -   ratioed_spectrum₁ is the ratioed spectrum from the first target        of interest    -   ratioed_spectrum₂ is the ratioed spectrum from the second target        of interest    -   spectrum₁ is the measured spectrum of the first target of        interest    -   spectrum₂ is the measured spectrum of the first target of        interest    -   spectrum₁₀₀ is the measured spectrum of the illumination source        The ratioed spectra are then used to illuminate an unknown        sample, e.g., a kidney. In other words, the spectra calculated        by equations (7) and (8) are used to illuminate the unknown        sample and the reflected illumination are measured with the        detector. Then, pixel by pixel, the second image, i.e., the        image obtained from the illumination of ratioed_spectrum₂, is        subtracted from the first image, i.e., the image obtained from        the illumination of ratioed_spectrum₁. The resulting image is a        chemically encoded image providing a quantitative assessment of        the targets of interest. It should be appreciated that each of        the foregoing measurements occur on a pixel by pixel basis and        therefore differences in illumination and/or pixel response can        be accounted for.

As described above, binning is the practice of merging charge from theadjacent pixels in a CCD prior to digitization in the on-chip circuitryof the CCD by specific control of the serial and parallel registers.Binning reduces the readout time and the burden on computer memory,increases the signal to noise ratio, but at the expense of imageresolution. To comprehend the binning practice in the Roper scientificcameras, consider the examples shown below in FIG. 21 (binning 1×1 whereno charges are summed provides the maximal resolution) against FIG. 4(binning 2×2 where charges from 4 neighboring pixels are summed). FIG.21(1) shows the CCD at the end of an exposure, wherein the capitalletters represent different charge accumulated on the CCD pixels.Readout of the CCD begins with the parallel readout phase. FIG. 21(2)shows simultaneous shifting of all pixels in a bottom row towards theserial register followed by the serial readout phase, FIG. 21(3) andFIG. 21(4) shown shifting of charge in the serial register into thesumming well which is then digitized. Only after all the pixels in thebottom row are digitized is the second row from the bottom moved intothe serial register. Thus, for example above, the order of shifting istherefore A1, B1, C1, D1, A2, B2, C2, D2, A3, . . . D6.

The charge that has integrated during the exposure is shown as capitalletters in FIG. 22(1). Readout begins with a parallel readout, as shownin FIG. 22(2); however, since binning of 2×2 is required, charge fromtwo rows of pixel, rather then a single row during 1×1 binning, isshifted into the serial register. Next, charge is shifted from theserial register, as shown in FIG. 22(3) and FIG. 22(4), two pixels at atime, into the summing well rather then a single pixel as in binning1×1. The result is that each readout event from the summing wellcontains the collected charge from four pixels on the CCD, i.e., asuperpixel. This procedure is iterated until the entire array has beenread and the formation of superpixels shown below for binning of 2×2.FIG. 23 shows similarly superpixel formations for 2×2 binning and 4×4binning.

In certain embodiments, hyperspectral data analysis provides spatiallydistributed contributions of oxyhemoglobin, which are obtained basedupon oxy- and deoxyhemoglobin reference spectrums. FIGS. 24A and 24Bshow a comparison of actual measured hemoglobin versus the predictedhemoglobin using pure hemoglobin samples and the actual measuredhemoglobin using the present invention. The system requires two or morereference spectra, followed by a least squares fit, at each pixel, ofthe measured spectrum by performing linear combinations of the referencespectra to obtain best fit curve. The resulting relative linearcontributions are scaled to produce a gray scale image.

Pure HbO₂ and Hb solutions were prepared at NIH by standard methodsusing blood collected from a healthy individual and reference spectrawere obtained from the original imaging system developed at NIH for theVisible region (400 nm-700 nm). A region of interest (520 nm-645 nm)containing the peaks for oxy- and deoxyhemoglobin was selected and usedfor imaging purposes in the visible region, see FIG. 24A. Similarmethods are used for HbO₂, Hb, H₂O, and lipids by scanning the NIRregions. References can be taken for HbO₂, Hb, HbCO and HbNO in thevisible regions. In addition, an unsupervised method can also be used,e.g., data can be pre-processed using a filter, and then normalizedthereafter. Here, each individual spectrum can be normalized first. Thetypical steps include, but are not limited to, measuring a spectrum,filtering the spectrum, normalizing the spectrum data, deconvoluting thespectrum data, and calibrating the corresponding data. The presentinvention does not concentrate or only measure the spectral amplitude;the present invention observes the changes in spectral broadening ornarrowing in addition to amplitude change.

In some embodiments, the protocol used may involve the following: Anexample involving seven subjects, for imaging the Human eye afterseeking approval from the Institutional Review Board, IRB. In thisexample, the sclera of the eye was illuminated using the light sourcefrom the slit lamp for five (5) seconds while acquiring theHyperspectral image cube. The imaging was achieved in two steps, firstlyon entering the lab, the subjects were asked to rest their chin on thechin rest of the slit lamp with closed eyes and light from the slit lampwas shone onto one of the closed eyelid which was brought in focus ofthe slit lamp. Secondly, upon getting the eyelid in focus using the slitlamp optics, the subjects were then asked to open their eyes, look inthe opposite direction from where the light was shone into the eye, andthe slit lamp was made to focus on the scleral vessels upon which,hyperspectral image data cube was obtained for five (5) seconds.

The setup for an embodiment of the present invention noninvasivenon-DLP® microscopic reflectance hyperspectral imaging system consistingof a source, optics, filter and detector as shown in FIG. 25. The slitlamp contains a 12V/30 W halogen source providing broad bandillumination, that illuminates the target and the reflected light fromthe target is guided by the microscopic optics onto a beam splitter, asemi-silvered mirror transmitting light to the eyepiece and to theattached hyperspectral imaging system. The radiation from the beamsplitter is directed towards the LCTF using relay optics. The LCTFcollects reflected light provided by the source into individual bandpasses of wavelengths that are placed on the focal plane array (FPA)detector with the use of a 60 mm Nikon lens fitted in front of the FPA.The FPA has an analog to digital, A/D, converter for digitizing thedata, which is transferred to a high end laptop PC for post processing.A computer program automatously manages the data collection bysynchronizing the timing between individual hardware components, forexample, tuning the LCTF and triggering the FPA, and setting parameters(e.g., image size, exposure time, spectral range and resolution, imagebinning, gain, filename) using a GUI that was built in V++. To sum upthe data acquisition, various parameters are inputted initially usingthe GUI developed in V++, after which the LCTF initializes and thedesired voltages calculated for the selected spectral range are storedin the palette. Upon initialization, the LCTF tunes to the firstwavelength specified in the palette, the camera is triggered, the CCD isexposed for the duration of the exposure time, and ADC digitizes theimage information storing it onto the laptop. This process is repeatedfor the remaining wavelengths. For the visible application, the focalplane array (FPA) used was the HQ2 CCD in combination with the visiblelow resolution LCTF. The raw hyperspectral image data was collected overa spectral range of 520 nm-602 nm with a spectral resolution of 2 nmincrements with the magnification set at 10× and CCD binning of 2×2 forincreasing readout speed. Thus an entire reflectance hyperspectral imagecube was acquired and saved to hard disk in around five (5) seconds.

The microscopic hyperspectral imaging system was then characterized. Thecharacterization of the system involved the characterizing the LCTF, theFPA and the slit lamp source. The LCTF was characterized for itsspectral band-pass, tune delay, and the tuning wavelength capability ofthe filter. The FPA was characterized for its ability to differentiatebetween objects, i.e., spatial resolution.

The spectral capabilities of the near infrared LCTF were determinedusing a calibrated Perkin Elmer Spectrometer. The spectrometer wasscanned in increments of 1 nm with the near infrared LCTF that was tunedto a specific wavelength placed in the collimated optical path of thespectrometer. This procedure was repeated for a series of sequentialwavelengths spanning the range of the near infrared LCTF, 650 nm to 1000nm, at every 50 nm and the corresponding transmittance spectra of theLCTF were noted. The transmission spectra from the spectrometer werethen analyzed in Matlab to measure the center wavelength the LCTFactually tuned to.

The desired wavelengths that were electronically sent to the LCTFcontroller are plotted on the X axis and the measured wavelengths, thespectrometer transmission spectral analysis to which the LCTF tuned, areplotted on the Y axis. Example data shows a linear regression curve fit(Y=1.00X−0.20, R2=1). From the example data, it is believed that anerror exists between the desired tuning and the actual tuning of theLCTF. To rectify this small error, a look-up table in the V++ softwarewas established from the relationship that was derived above which makessure that the LCTF gets tuned to the wavelength which the operatordesires.

The visible low resolution LCTF similarly was calibrated, however, ahigh resolution LCTF was used having a band-pass lying between 0.19 nmat 500 nm and 0.75 nm at 700 nm. As shown in FIG. 26, the input lightfrom the source is passed through the low resolution LCTF, which is tobe calibrated, tuned to a particular wavelength onto the high resolutionLCTF which is scanned with an increment of 1 nm over its spectral rangeof 480 nm-720 nm, and focused using the 60 mm Nikon camera lens onto theCCD camera generating a series of images, i.e., a hyperspectral imagecube. This procedure was repeated until the low resolution LCTF wastuned in the wavelength range 500 nm-700 nm at every 10 nm and ahyperspectral cube captured. The hyperspectral cubes generated were thenanalyzed in Matlab to locate the center wavelength the LCTF was actuallytuned to. As in the near infrared LCTF calibration plot, here also thedesired wavelengths sent to the LCTF controller were plotted on the Xaxis and the measured wavelengths from the hyperspectral analysis wereplotted on the Y axis, see FIG. 27. Points in FIG. 27 are example valuesand the dashed line represents a linear regression curve fit(Y=0.9984X−0.5013, R2=0.9999). From this relationship, a small look-uptable was generated in the V++ software tuning the LCTF to the desiredwavelength that the operator desires.

Similarly the spectral band-pass for the visible low resolution filtervaries from 3.7 nm at 400 nm to 12.2 nm at 700 nm, see FIG. 28. Theconclusion from these Figures is that the LCTF band-pass is wavelengthdependent. This bandwidth is more than sufficient to spectroscopicallyresolve HbO₂ and Hb which have spectral characteristics that are severaltimes broader than the broadest band-pass.

In the hyperspectral imaging system images are collected sequentially atwavelengths differentiated continuously by the LCTF that tunes from onewavelength to another. LCTF takes typically 50 to 150 ms to switch fromone wavelength to another, i.e., the response time. Any imagingapplication collecting sequential image data through the LCTF mustaccount for the response time required to tune the LCTF. In the V++program developed to control the acquisition of the hyperspectralimaging system, a tune-delay time is introduced that accounts for theLCTF tune time. An example was done to determine this tune-delay ortune-wait time for the LCTF's; the setup for which is shown in FIG. 29.In this example, light focused on a spectralon target is reflected ontothe FPA through the LCTF and a hyperspectral cube was collected.Hyperspectral cubes were collected for varying tune-delay time startingfrom 0 ms to 100 ms in increment of 10 ms.

Similarly, spectral analysis was performed on the visible LCTF, and asshown in FIG. 30 there is little or no difference between 0 ms, 30 msand 50 ms, thus suggesting a tune-delay of 0 ms for the visible LCTF.

The tune-delay time increases the exposure time for collecting thehyperspectral, hence keeping it to a minimum is advantageous.Tune-wait/Tune-delay time depends on various factors such as the liquidcrystals and the electronic components in the circuitry. Thus, tune-waittime is typical for each LCTF.

The following describes the calibration of the FPA. The spatialresolution of the microscopic hyperspectral imaging system wascharacterized, which is defined as the ability to distinguish betweentwo closely spaced objects within an image. The spatial resolution ofthe system was established by computing the percent contrast whichdepends on various factors such as the focal plane array, filter, slitlamp magnification, type of camera lens, f-stop, depth of field, anddegree of pixel binning. Modifications of any of the factors alter thespatial resolution of the system. In view of the foregoing, spatialresolution was evaluated with various degrees of binning andmagnification. The percent contrast, C, was determined using theexpression equation (9) shown below, and plotted as a function ofspatial resolution in millimeter.

$\begin{matrix}{C = {\left( \frac{I_{\max} - I_{\min}}{I_{\max} + I_{\min}} \right) \times 100}} & (9)\end{matrix}$

where:

-   -   I_(max) is the maximum intensity reflected by a line of the        resolution    -   target (white bar) of the resolution target are shown in FIG.        31; and,    -   I_(min) is the minimum intensity from the nonreflecting area        between the lines (dark bar) of the resolution target are shown        in FIG. 31.

A 1951 quartz USAF resolution target is depicted in FIG. 31 where Adepicts a portion of the target and the corresponding reflectedintensity taken along a row of pixels is depicted in B was used todetermine the spatial resolution. It should be appreciated that a linepair is defined as the distance between the first edge of a white barand the second edge of a dark bar. Generally, the spatial frequency, ornumber of line pairs, increases (the number of pairs/unit length) as thepercent contrast decreases. The quartz target was placed in the imagingpath and a single image was collected which was then analyzed todetermine the percent contrast by using equation (7). This procedure wasfollowed for determining spatial resolution of the near infrared imagingas well as visible imaging system. Binning and magnification werechanged keeping other parameters the same, and images were collected andanalyzed to obtain the percent contrast. Percent contrast calculationswere done for the near infrared microscopic hyperspectral imaging systemand plotted as a function of spatial resolving power (mm) in FIG. 32.The graph of FIG. 32 includes percent contrast as the dependent variable(y-axis) and spatial resolution in millimeter as the independentvariable (x-axis) and it has been determined from the regression curvefit models when various binning embodiments cross the 20% contrastthreshold set by the Rayleigh criterion, i.e., the horizontal dashedline.

Percent contrast of the visible microscopic hyperspectral imaging systemis plotted as a function of spatial resolving power (mm) in FIG. 32.Each embodiment shown in FIG. 32 has a polynomial regression curve fitmodel. For a magnification of 10× and binning of 1×1 (diamond shapeddata points), the regression equation is y=−996.04x2+548.86x+5.7296.Keeping the magnification same but changing the binning to 2×2 (squareshaped data points), the regression equation isy=−919.92x2+544.64x+0.8112. For a binning of 3×3 (triangle shaped datapoints), and 4×4 (cross shaped data points), the respective regressionequations are y=−660.48x2+498.39x−4.0232 and y=−531.26x2+458.81x−5.5901.Again, in all the above embodiments, the dependent variable (y-axis) ispercent contrast and the independent variable (x-axis) is spatialresolution in millimeter, and it is determined from the regression curvefit models when it crosses the 20% contrast threshold set by theRayleigh criterion, i.e., the horizontal dashed line. The spatialresolution for the visible hyperspectral imaging system with themagnification at 10× is 0.027 mm at binning 1×1, 0.038 mm at binning2×2, 0.052 mm at binning 3×3 and 0.06 mm for binning 4×4. As mentionedabove, the binning decreases the spatial resolution which is verified bythe spatial resolution numbers obtained.

Another interesting comparison was made using the visible microscopichyperspectral imaging system by obtaining percent contrast measurementsfor the visible system without the presence of the relay optics in thelight path. These measurements are plotted in FIG. 33. Regressionequations were obtained and analyzed keeping the Magnification at 10×,and changing the binning on the FPA to find out the spatial resolutionfor the system without the relay optic, yielding spatial resolutions of0.072 mm for binning of 1×1, 0.104 mm for binning of 2×2, 0.129 mm forbinning of 3×3 and 0.169 mm for binning of 4×4. The spatial resolutionfor the system without the relay optics was found to be worse then thatfor the system with the relay optics in the light path. The relay opticswere introduced in the system to increase the magnification of thesystem and allow the system to occupy a larger area of the CCD chip thusimproving the overall spatial resolution of the system as verified bythe spatial resolution numbers for the system with and without the relayoptics.

The following describes the system performance of an embodiment of thepresent invention. Hyperspectral imaging is emerging as a powerfulimaging tool to exemplify spectral as well as the spatialcharacteristics. An embodiment of a present invention microscopic nearinfrared hyperspectral imaging system was used to image, non-invasively,microvascular perfusion of the dermal tissue, in vivo, for assessingoxyhemoglobin contribution during a resting condition for ten subjects.The hyperspectral data deconvoluted for the contribution ofoxyhemoglobin is gray scale encoded, meaning the greater the pixelintensity the greater the oxyhemoglobin contribution as depicted by thegray scale bar associated with an image.

The microscopic hyperspectral system was further applied towards imagingthe microvasculature structure present in the anterior region of thehuman eye, especially the sclera, which required imaging in the visiblespectrum, i.e., 520 nm-602 nm. A hyperspectral image cube was obtainedusing the visible slit lamp microscopic imaging system, with parameterssuch as setting the magnification to 10×, setting the binning to 2×2 andvarying the wavelength from 520 nm-602 nm, incremented at 2 nm. Theseparameters successfully reduced the acquisition time for thehyperspectral image cube to within 5 seconds which is critical forimaging the eye since the subjects involved in the data collection arenot allowed to blink or move their eye since that would create artifactsin the images.

The hyperspectral cube was deconvoluted for oxyhemoglobin contributionvalues that are color encoded and spatially depicted in FIG. 34A. Asmall region over a high oxyhemoglobin contribution region, shown withinthe small black box, as suggested from the color bar, i.e., red areas,was selected and pixels under the region were averaged to obtain aspectrum shown in FIG. 34C which resembles the oxyhemoglobin spectrum.Similarly, the spectrum under the area of the small pink box wasselected and pixels under the region were averaged to obtain a spectrumshown in FIG. 34B, and represents a deoxyhemoglobin spectrum.

Seven healthy subjects were imaged to obtain scleral tissueoxyhemoglobin contributions for assumed arterial and venous structuresin the image as displayed by the spectroscopic information. The averageoxyhemoglobin contribution values for these seven subjects wereobtained. The regions from under which the above average valuesevaluated were not random but visually selected depending on thespectroscopic information the region yielded.

Qualitative values with quantitative evaluations pending were obtained.These values demonstrate that there exist marked differences betweenarterial and venous structure oxyhemoglobin contributions and that thesystem is capable of distinguishing between arterial and venousstructure in the tissue being imaged.

The following description sets forth embodiments of a present inventionmicroscopic visible hyperspectral imaging systems. After demonstratingcapabilities of the microscopic hyperspectral imaging system usingdermal microvasculature imaging, other applications of the presentinvention are now discussed, in particular, imaging of the human eye,especially the sclera. The sclera is the white portion of the eye,covered by the episclera, tenon's capsule, and the conjunctiva. Theblood vessels seen on the scleral surface are actually found betweenthese various regions. Scleral disease such as scleritis andepiscleritis, are uncommon in opthalmology patients, however theypresent symptoms for serious, painful and life threatening systemicdiseases such as rheumatoid arthritis, syphilis, spondylitis, etc.

The anterior region of the human eye was imaged using the microscopicvisible hyperspectral imaging system. The visible system uses theCoolSNAP_(ES) CCD in conjunction with the visible low resolution LCTFand the slit lamp microscope. The CoolSNAP_(ES) CCD's spatial resolvingpower at a binning of 2×2 and a magnification of 10× is 0.038 mm, whichis better then the resolving power of the PIXIS 400 BR with the sameparameters, i.e., 0.083 mm. Since speed is an important trade off withthe imaging of the eye, the 20 MHz readout rate is preferred against 2MHz readout rate of the PIXIS FPA. Also to decrease the acquisition timeof the system the spectral range and spectral resolution were reduced to520 nm-602 nm at increments of 2 nm, instead of the previously reportedmacroscopic version of the visible hyperspectral system that used aspectral range of 520 nm-645 nm, at increments of 1 nm. These were thetwo major trade offs considered to reduce the data acquisition time to 5seconds, which acquisition time is clinically suitable for eye imagingapplications. Inputting these example parameters with the slit lampmagnification set to 10×, the sclera of the subject was imaged to obtaina hyperspectral data cube that upon deconvolution with the visible rangereference spectra of oxyhemoglobin and deoxyhemoglobin rendered an imagethat contained spatially distributed contributions of oxy-hemoglobin.

FIG. 35 is an illustration of the basic setup of a DLP hyperspectralimaging system of the present invention. The setup for noninvasive DLP®microscopic reflectance hyperspectral imaging system consisting of aspectrally-tunable source, optics, a detector, a computer, and a DLP®.The radiometric radiation source can be a 12V/30 W halogen source and/ora QTH (quartz tungsten halogen) source providing broad bandillumination. The radiation is directed towards the DLP® using relayoptics. The DLP® differentiates the broadband reflected light intoindividual band passes or provides a spectrum of wavelengths and isdirected to the object to be analyzed. Major components include, but notlimited to: white laser source 350 and spatial light modulator 352, suchas a DLP®, used to create one or more light wavelengths or a bandspectrum which are captured as raw images 354. Raw images 354 are thenprocessed to detect known chemometrics 356, which are then output asvisual image 358. The user is then able to determine if they wish tosave or discard the image, which image may be stored on storage device360, e.g., hard drive, CD-ROM, etc.

Another example of the present invention is demonstrated as follows. Thehyperspectral slit imager acquisitions hyperspectral data that is grayscale encoded as the percentage of oxyhemoglobin at each pixel of theimage along with its associated spectrum. Spectra were selected fromdarker pixels indicating a venous structure, the purple spectrum, versusbright pixels indicating an artery, the red spectrum, as shown in FIG.36. FIG. 36 demonstrates retinal imaging of oxyhemoglobin contributionusing visible reflectance hyperspectral imaging system. Other part ofthe eye, for example the conjunctiva, can also be imaged and can also becolor encoded, the redder the pixel, the greater the percentage ofoxyhemoglobin, and a variety of image processing techniques may beapplied, for example, feature extraction or optical biopsy, for futureevaluation (see FIG. 37). FIG. 37 is a hyperspectral imaging of smallvessels within human conjuctiva of the eye.

FIG. 38 is an image taken using one embodiment of the present inventionby using a DLP system with a “2 shot” illumination scheme. The resultswere obtained using illumination with a spectrum of light. Thisembodiment includes illumination with the difference of two (2)principle component spectra (one spectrum that is consistent with thepositive values and the other in which the negatives are multiplied bynegative one, illuminated and then the image multiplied by negative oneagain). The resulting images are then normalized and processed.

In another embodiment, the present invention includes using opticalfilters as a method for fluorescing indocyanine green dye (ICG). Theembodiment enables the identification of organs such as the biliarystructures when ICG is injected into a blood vessel such as the femoralartery of an animal or human. This embodiment enables fluorescence ofbiliary structures. Here, the illuminator can be a Texas Instrument'sDLP illuminator and the fluorescence image of the ICG can more preciselyilluminate the exact spectrum for maximizing the fluorescence of the ICGand imaging it in the biliary structures. The present Applicants havecompleted the optical filter system and have successfully imaged ICG andbile fluorescence in capillary tubes. This embodiment can further bemodified to be used in animals and humans or clinical trials.

In certain embodiments, the present invention can be used in conjunctionwith using a supercontinuum laser instead of a plasma source toilluminate the DMD chip. The laser used in this embodiment does providegreater intensity, smaller band passes, and larger spectral range.Supercontinuum or white-light laser sources are recognized in the art.Briefly, they are typically suitable for applications in spectroscopyand microscopy. It can include a pump laser and a microstructured fiber(either a photonic-crystal fiber or a tapered fiber). Although 80 fspulses from a Ti:sapphire laser and 200 fs pulses from an ytterbium(Yb):glass laser have been used to generate supercontinuum light,researchers have demonstrated a more-compact and lower-cost portablefemtosecond supercontinuum source with a small footprint. Moreover, inother embodiments, the present invention can include a light emittingdiode (LED) based illumination source. The source may include a singleLED or a plurality of LEDs.

In some embodiments, the present invention includes image analysis of apatient wearing a silk gown. Near infrared light is known to see throughsilk. Wearing a silk gown allows the patient to cover them selves whileletting the NIR camera see through to the tissue for analysis. In thisembodiment, the patient can lay on a piece of clear material, e.g.,plexiglass or glass. A layer of oil can help negate index or refractioneffects and improve imaging similar to an oil immersion microscopeobjective lens. This embodiment provides maximum privacy to the personbeing imaged, preserves body heat and provides accessibility for thesurgeon.

In another embodiment, the present invention can also include at leastone deconvolution algorithm. The algorithm is typically used fornormalizing each spectrum at each pixel as a step in the pre-processing.This algorithm is counter-intuitive to the skilled artisan in the art orfor the general spectroscopist. The skilled artisan would not considerand would question this unique and unexpected algorithm.

In yet another embodiment, the present invention can be used duringkidney surgery. The kidney is highly vascularized and tends to bleedcopiously during surgery, as such, surgeons minimize this bleeding byclamping off the blood supply. However, clamping off this organ alsolimits greatly the time permitted to perform the surgery withoutaffecting long-term kidney function. The present invention allows asmall amount of blood to leak past the clamp to perfuse the kidney,thereby giving the surgeon more time to perform the surgery beforepermanent kidney tissue damage takes place. The hyperspectral imager ofthe present invention aids in visualizing and determining when thekidney is at risk of permanent damage before damage occurs by providingthe surgeon real-time information regarding kidney cell and tissuestatus.

The present invention can be applied in various field, e.g., incholecystectomy, amputations, burns, skin flap evaluation, visualizingareas of angiogenesis, probes that bind antigens and absorbs NIR duringpathological evaluations and in vivo, quality control ofpharmaceuticals, monitoring vascular changes and drug discovery inresponse to pharmaceuticals, monitoring diabetic retinopathy, diseasessuch as cancer, diabetes, sickle cell, anemia, bilirubin, raynauds,ulcers, burns, skin flaps, surgery, gallbladder, brain, monitoring woundhealing, and early detection of wound infections.

FIG. 39 is a graph that shows one hundred twenty six (126) separatewavelengths using a LCTF to separate the bands prior to illumination,after which an image is captured with each bandpass or with a singleframe including up to one hundred twenty six (126) images. FIG. 40 is agraph that shows a band spectrum in which a digital micromirror arraywas used to create spectral illumination that allows for a lower numberof images per frame. FIG. 41 shows a comparison of data obtained withthe LCTF and the digital micromirror array illumination using the singlebandwidths of FIG. 39. FIG. 42 shows a comparison of data obtained withthe LCTF and the digital micromirror array illumination using the singlebandwidths of FIG. 40.

FIG. 43 is a flow chart of the basic “2-shot” algorithm. FIG. 44 is aflow chart of the processing of the data cube obtained using the basic“2-shot” method.

FIG. 45 is a flow chart of the acquisition method of the basic “2-shot”method. Briefly, normalized absorption spectra in the 520 nm-645 nmwavelength range for HbO₂ and Hb are subtracted from each other, and thepositive areas become the two illumination spectra. The relativeintensity of each illumination spectrum is stretched from 0 to 100 tomaximize the overall light intensity and match the required OL-490 inputformat. Each data cube (M×N×2) consists of one M×N pixel image taken inthe first image capture and one M×N pixel image or shot captured, takenfor the second image or shot captured.

FIG. 46 shows sample images obtained for finger occlusion at differenttimes using the present invention. FIG. 47 compares the images obtainedusing visible light and near infrared (NIR) of the reperfusion of a footfollowing removal of the shoe. FIG. 48 shows the in vivo hyperspectralimaging of human tissue, spatial variation of percentage of HbO₂ andsurface temperature in response to a burn.

The present invention describes a “3 shot” illumination method in anintegrated DLP hyperspectral imaging (HSI) system which reduces thenumber of image frames required to generate a processed image that iscolor-coded based on matching the reflectance of each pixel in an imageto known reflectance spectra. The “3 shot” method of the presentinvention can replace traditional hyperspectral imaging which requiresmany frames to be acquired at contiguous wavelengths throughout awavelength range of interest.

In the “3 shot” method of the present invention, a sample is illuminatedwith three complex spectra which have varying levels of intensity ateach wavelength across the wavelength range of interest. Traditionalhyperspectral imaging captures images for discrete wavelengths withnarrow bandwidths. Contrarily, in the “3 shot” method, a series ofnarrow bandwidths is replaced with a broadband spectrum, in which therelative intensity of each wavelength is attenuated according toreference spectra. Without explicitly measuring the reflectance spectrumof a tissue, the “3 shot” method implicitly determines how closely thesample matches one of a pair of reference spectra.

After illuminating a sample with three broadband components of referencespectra, the “3 shot” method of the present invention subtracts anddivides the resulting three images, and outputs several processed imagesper second. Present technology, on the other hand, illuminates orfilters multiple narrow bandwidths, measures the sample reflectance foreach of those discrete bandwidths, compares the reflectance spectrum toknown reference spectra, and outputs, at most, one processed image perseveral seconds.

The advantages of the “3 shot” method of the present invention overtraditional hyperspectral imaging makes the present invention ideal forreal-time visualization of the molecular components of tissue forclinicians and surgeons. In addition the “3 shot” method of the presentinvention can be implemented in any hyperspectral imaging device whichhas the ability to illuminate with a variable intensity spectrum.

The present invention describes a “3 shot” illumination method in anintegrated DLP HSI instrument. The “3 shot” method generates an image ofthe target very quickly in comparison to the more commonly used andpreviously described spectral sweep illumination method.

The data acquisition software automatically tune the DLP® technology andtrigger the camera for collecting a series of spectroscopic imagesformatted as a hyperspectral image cube. Next, the spectroscopic imagedata are deconvoluted using chemometric analysis methods. The resultinggray scale or color encoded images provide the clinician with anon-invasive visualization of the chemical state within themicro-vasculature perfusing the tissue while the patient is in theclinic or surgery.

Hyperspectral imaging is the equivalent of reflectance spectroscopy atmany discrete points arranged in a spatial pixel array. Reflectancespectroscopy involves collecting the reflectance of light at eachwavelength, in the spectral dimension. Projecting the spectraldimensions to each pixel in an image results in a 3D data cube thatcontains the intensity spectrum for each pixel in a 2D spatial array. Ahyperspectral data cube can be acquired in two fundamental ways:scanning a point spectrometer over a spatial area, or sweeping thewavelengths of light incident on an array detector. With the2-dimensional array detectors available today, the reflectance of lightcan be collected in two spatial and one spectral dimension.

The first way to capture a hyperspectral image cube is by moving asample on a mechanical stage to raster-scan a point spectrometer throughthe image field of view (FOV). (See Sellar R G, Boreman G D,Classification of imaging spectrometers for remote sensing applications.2005; 44:1-3). A point spectrometer couples a very small area ofbroadband light into a prismatic grating. The grating spatiallydisperses polychromatic light into its monochromatic components, whichare then uniquely detected to quantify the intensity spectrum as afunction of wavelength. The spectrum measured by a point spectrometer isa measurement of all the light coupled into the spectrometer. Togenerate a 3D hyperspectral image cube, the point spectrometer mustcapture an intensity spectrum at each spatial pixel in the image FOV.

The second fundamental method of hyperspectral imaging is acquiring allspatial information at a wavelength of interest and scanning through aspectral range using optical or electro-optical filters. A liquidcrystal tunable filter (LCTF) or acousto-optic tunable filter (AOTF) isused to cut off all broadband light except for a precisely tuned narrowbandpass. (See Sellar R G, Boreman G D, Classification of imagingspectrometers for remote sensing applications. 2005; 44:1-3). Ascientific grade camera captures an image of a full 2D scene (e.g.,tissue sample) at each narrow bandpass through a sweep of contiguouswavelengths, resulting in a 3D hyperspectral image cube.

The instrumentation in the present invention consists primarily of aspectral light engine with DLP® technology providing the spectroscopicillumination, a digital camera with a scientific grade CCD for imaging,and software designed and developed to manage the data acquisition andthe chemometric visualization. The hyperspectral imaging systemcomprises an OL 490 spectral light engine created with TexasInstruments' “Digital Light Processor” (DLP®) technology. In the OL 490,polychromatic visible light is dispersed onto the micromirror array of aDLP chip so that each column of micromirrors corresponds to a narrowband of monochromatic light. Programming the mirrors individually allowsthe user to precisely define the intensity of each wavelength in thelight engine's optical output spectrum. The OL 490 can illuminate withnarrow bandpasses of light as in traditional hyperspectral imaging, butthe novelty of the DLP HSI is using the OL 490 to illuminate withcomplex broadband illumination spectra. Regardless of the illuminationscheme, the light from the OL 490 is reflected from a tissue sample ofinterest and detected by a scientific grade CCD focal plane array. Thedata acquisition software automatically tunes the DLP® technology andtriggers the camera for collecting a series of spectroscopic imagesformatted as a hyperspectral image cube. Then, the spectroscopic imagedata are deconvoluted using chemometric analysis methods. The resultinggray scale or color encoded images provide the clinician with anon-invasive visualization of the chemical state within themicro-vasculature perfusing the tissue while the patient is in theclinic or surgery. A simplified schematic of the instrumentationdescribed above is presented in FIG. 49.

One of the many goals of the present invention is to demonstrate thatthe “3 shot” illumination method in a DLP®-based visible hyperspectralimaging system can be used routinely for a variety of medicalapplications. The speed afforded by the “3 shot” illumination method ofthe present invention reduces the current acquisition time from minutesto seconds making hyperspectral imaging a practical everyday surgicaland clinical tool for imaging.

The DLP® HSI with “3 shot” illumination and chemometric deconvolutioncan be applied to all products that use light, for example, the fieldsof clinical endoscopy, clinical chemistry, microscopy, surgicalmicroscopy, drug discovery, microarray scanners and microplate readers.

As described above, the present invention can be used during kidneysurgery. The kidney is highly vascularized and tends to bleed copiouslyduring surgery, as such, surgeons minimize by clamping off the bloodsupply. However, clamping off this organ also limits greatly the timepermitted to perform the surgery without affecting long-term kidneyfunction. The present invention allows a small amount of blood to leakpast the clamp to perfuse the kidney, thereby giving the surgeon moretime to perform the surgery before permanent kidney tissue damage takesplace. The hyperspectral imager with “3 shot” illumination of thepresent invention aids in visualizing and determining when the kidney isat risk of permanent damage before damage occurs by providing thesurgeon real-time information regarding kidney cell and tissue status.

Currently there are three illumination methods implemented in the DLPHSI for hyperspectral imaging: Full Spectral Sweep, Spectral Sweep, andthe “3 Shot” method described in the present invention. Theseillumination methods will be discussed in the remaining sections;however, new illumination methods are easily created by programming thespectral light engine. In fact, the integrated DLP HSI system of thepresent invention has the ability to change the illumination spectrum toany imaginable narrow or broadband spectrum in the wavelength range ofthe OL 490. This versatility means that the DLP HSI can be used for anynumber of imaging applications.

The full spectral sweep method is now described. To utilize thepractical wavelength range of the system, the full spectral sweep methodsweeps bandpass illuminations having 10 nm bandwidths from 450 nm to 650nm in 4 nm increments. A hyperspectral image cube acquired with thismethod consists of 51 slices and is generally used to explicitly measurethe absorbance spectroscopy of an area of interest. Formally, thefollowing series of illuminations is programmed into the OL 490:{L _(1,FullSweep) =L _(OL490)(λ₁),L _(2,FullSweep) =L _(OL490)(λ₂), . .. L _(51,FullSweep) =L _(OL490)(λ₅₁)}  (10)for,λ_(n)=450+4(n−1),n={1:51}  (11)where, L_(OL490)(λ_(n)) is the total radiant power emitted from the OL490 at a center wavelength, λ_(n), with a 10 nm full width half maximum(FWHM) bandwidth.

After proportioning with the background, each slice in the cuberepresents the absorbance of the bandpass illumination at which theimage is acquired. Thereby, each spatial pixel in the processed cuberepresents a discrete absorbance spectrum of the tissue or othermaterial in the physical position corresponding to that pixel. Thewavelength limits of the full spectral sweep method intentionallytruncate the wavelength limits of the OL 490 since the output intensityof the OL 490 is very low near the extremes of its wavelength range.

This illumination method is only used to help calibrate the system as aspectrophotometer, and no algorithms currently exist to process itsimage cubes for visualization of tissue chemistry. However, with thespectral information acquired using full spectral sweep illumination, itis possible to search for spectral signatures of chemical chromophoresand develop new processing algorithms.

The spectral sweep method is now described. To mimic the illumination ofLCTF-based hyperspectral imaging systems, the spectral sweep methodsweeps bandpass illuminations with 10 nm bandwidths from 520 nm to 645nm in 1 nm increments. A hyperspectral image cube acquired with thismethod consists of 126 slices and explicitly measures the absorbancespectroscopy of an area of interest. Formally, the following series ofilluminations is programmed into the OL 490:{L _(1,Sweep) =L _(OL490)(λ₁),L _(2,Sweep) =L _(OL490)(λ₂), . . . L_(126,Sweep) =L _(OL490)(λ₁₂₆)}  (12)for,λ_(n)=520+(n−1),n={1:126}  (13)where, L_(OL490)(λ_(n)) is again the total radiant power emitted fromthe OL 490 at a center wavelength, λ_(n), with a 10 nm FWHM bandwidth.

After proportioning out the background, each slice in the cuberepresents the absorbance of the bandpass illumination at which theimage is acquired. Thereby, each spatial pixel in the processed cuberepresents a discrete absorbance spectrum of the tissue or othermaterial in the physical position corresponding to that pixel, identicalto the full spectral sweep method, but with better spectral resolutionand narrower wavelength range. Each pixel absorbance spectrum is thencompared to reference spectra for oxy-hemoglobin (HbO₂) anddeoxy-hemoglobin (Hb) in FIG. 50A by multivariate least squares analysisto calculate the percent HbO₂ for that pixel. The output bitmap, afterprocessing with ‘Oxyz—Jet’, is a color-coded two-dimensional image,where more intense pixels (red) signify that the pixel absorbancespectrum more closely resembles the HbO₂ reference and less intensepixels (blue) signify that the pixel absorbance spectrum more closelyresembles the Hb reference.

Spectral sweep illumination is slow for two fundamental reasons:acquisition of 126 frames takes 126 times longer than acquisition of asingle frame and processing of such large data cubes takes considerabletime. One suggested solution to this speed issue has been selecting onlya few narrow illumination bands to significantly reduce the number ofslices in the cube, thereby reducing the acquisition time and processingtime. (See Guo, B, Gunn, S R, Damper, R I, Nelson, J D B, Band selectionfor hyperspectral image classification using mutual information. 2006;3:522-526; and, Du, Z, Jeong, M K, Kong, S G, Band selection ofhyperspectral images for automatic detection of poultry skin tumors.2007; 4:332-339). This solution has been shown to speed up dataacquisition and processing, but does not necessarily gather the samespectral information as a spectral sweep.

The present invention “3 shot” method is now described. The “3 shot”method of the present invention enables the gathering of spectralinformation equivalent to the spectral sweep method with fewer imagesbeing captured. The “3 shot” method sequences through three complexbroadband illumination spectra in the 527 nm to 638 nm wavelength range.A hyperspectral image cube acquired with this method consists of three(3) slices and is used to calculate the percent HbO₂ for each spatialpixel without explicitly measuring the absorbance spectra. Formally, thefollowing illuminations are programmed into the OL 490:

$\begin{matrix}{L_{1,{3\mspace{11mu}{shot}}} = {\sum\limits_{n = 8}^{119}{{L_{{OL}\; 490}\left( \lambda_{n} \right)}{V_{1}\left( \lambda_{n} \right)}}}} & (14) \\{L_{2,{3\mspace{11mu}{shot}}} = {\sum\limits_{n = 8}^{119}{{L_{{OL}\; 490}\left( \lambda_{n} \right)}{V_{2}\left( \lambda_{n} \right)}}}} & (15) \\{L_{3,{3\mspace{11mu}{shot}}} = {\sum\limits_{n = 8}^{119}{L_{{OL}\; 490}\left( \lambda_{n} \right)}}} & (16)\end{matrix}$For the wavelengths described by equation 8, V₁(λ_(n)) represents thepositive subtraction of normalized reference spectra and for thewavelengths described by equation 9, V₂(λ_(n)) represents the negativesubtraction of normalized reference spectra as shown in FIG. 50B. Thelimits of summation are intentionally n=8 and n=119, because thereference spectra have been filtered by a moving average filter thattrims seven discrete values from the beginning and end of each spectrum.

FIG. 51 is a block diagram that shows the experimental procedure used tocapture images using the “3 shot” method of the present invention andthe MATLAB® algorithm used to process the images captured using the “3shot” method of the present invention.

FIGS. 52A-52C are an illustration of the “3 shot” illumination method ofthe present invention used for visualizing blood oxygenation. FIG. 52Ashows the normalized absorbance spectra in the 520 nm-645 nm wavelengthrange for HbO₂ and Hb subtracted from each other, where the positiveareas become the two illumination spectra. FIG. 52B shows that therelative intensity of each illumination spectrum is stretched from 0 to100 to maximize the overall light intensity and match the requiredOL-490 input format. FIG. 52C shows the absorbers and scatterers thatare not HbO₂ and Hb.

After proportioning out the background, each slice in the cuberepresents the absorbance of the broadband illumination at which theimage is acquired. Absorbance spectra are not explicitly measured, but amathematic combination of the three slices results in a singletwo-dimensional image where higher pixel values indicate absorbancesimilar to HbO₂ and lower pixel values indicate absorbance similar toHb. The data cube is forty two (42) times smaller than the spectralsweep data cube, the processing algorithm is considerably simpler, andthe resulting visualization of tissue oxygenation is the same. Thus, acolor-coded bitmap image that appears nearly identical to one generatedby the spectral sweep method can be generated by the “3 Shot” methodmuch more quickly.

Three complex broadband illumination spectra were derived from thereference absorbance spectra of Hb and HbO₂ for the “3 Shot” methoddescribed in the present invention. Initially the OL 490 was programmedto illuminate with these spectra by directly copying the spectra intoGwectra files. The optical output measured by a spectrometer when thesespectra were explicitly programmed is seen in FIG. 53A. Explicitly usingthe calculated spectra, the measured optical output was significantlydifferent than the desired illumination. In the plot, the solid line(measured optical output) is shifted horizontally from the dotted line(desired optical output), and the measured first peak is not as intenseas the desired first peak.

In order to generate an illumination spectrum that more closely matchesthe desired optical output, a new Gwectra file was written. In the newfile, the intensities at each wavelength were shifted horizontallyaccording to the center wavelength calibration curves mentioned earlierand vertically according to the error between the measured and desiredintensities. Programming the OL 490 with the new Gwectra file, theoptical output is seen in FIG. 53B. In the plot, the solid and dashedlines (measured optical output and desired optical output, respectively)coincide much better than in the previous plot.

Reprogramming the mirrors of the DMD in the OL 490 allows for refinementof illumination spectra for the “3 Shot” method. The same wavelengthcalibration and intensity adjustment was performed for the other twocomplex broadband illuminations. For most of the wavelength range, theactual optical output matches well with the desired optical output. Dueto the minimum bandpass limitation when using the 350 μm slit, it isfundamentally impossible to mimic the two bumps desired between 620 nmand 640 nm. If lower overall light intensity is adequate, the 150 μmslit, which has a smaller minimum bandpass limitation, can be used tofurther refine the actual optical output.

Speed is a key requirement in the DLP hyperspectral imaging systemdesign. Camera selection is often based primarily on speed. The purposeof the “3 Shot” method of the present invention is speed. One of themain reasons hyperspectral imaging is not a primetime medical imagingmodality is its inherent lack of speed. The DLP HSI prototype of thepresent invention is purposefully versatile. The user can adjustexposure times (constant or variable), binning of the CCD array,processing algorithms for chemometric visualization, lens aperture, andOL 490 illumination methods. To address the need for speed, it isimportant to understand the role each of these parameters play indetermining the final frame rate of the system.

Several fundamental bottlenecks limit the maximum speed of the DLP HSI.The minimum exposure time of the OL 490 is 80 μs, the minimum exposuretime of the HQ2 is 210 μs, the maximum transfer rate of the camera tothe computer is 20 MHz, and the image processing time is significant.

Theoretically, increasing exposure time in each slice will increaseacquisition time for each image thereby increasing total acquisitiontime for the hyperspectral cube. Practically, exposure time is varied toachieve near 12,000 counts at the maximum intensity when capturing abackground cube, so exposure time is not considered an independentvariable. Rather, exposure time depends on lens aperture, camerabinning, gain, illumination intensity, and focal distance. Illuminationintensity is dependent on illumination method and is fixed for eitherthe spectral sweep illumination method or the “3 shot” illuminationmethod of the present invention. Focal distance is variable and isadjusted depending on the physical constraints of the clinical orsurgical setup. In surgery, the camera must be at least 1 m away fromthe subject and should be unobtrusive to the surgeon. On the lab bench,the camera is set to a focal distance of 45 cm. Camera gain can bevaried according to CoolSNAP HQ2 specifications while the lens aperturecan be varied from f/16 to f/1.4. The two parameters in concert affectthe exposure time primarily, with side effects of focus range anddetector noise. For the following speed tests, ‘high gain’ refers to acamera gain of 3 and a lens aperture of f/1.4, while ‘low gain’ refersto a camera gain of 1 and a lens aperture of f/8. Camera binning affectsboth exposure time and spatial resolution. For the following speedtests, bins of 2×2 and 4×4 are set. In the clinic, ‘low gain’ and 4×4binning is typically used to ensure good quality imaging at reasonablespeeds.

A final parameter that does not affect exposure time but still affectsthe speed of the system is the processing algorithm. The most basicprocessing algorithm reads the hyperspectral image cube into MATLAB andoutputs one of the raw images as a Windows bitmap file for display. Forvisualization of oxygenation, “Oxyz Jet” and “3shot Jet—*,” where * canbe any pair of thresholds, are the processing algorithms used. In eachof the oxygenation algorithms, the hyperspectral image cube andbackground cube in the same parent directory are read into MATLAB forcalculating absorbance. Chemometric analyses transform the absorbancecube into a chemically relevant image which is output as a bitmap fordisplay.

To measure the effect of each parameter on the overall speed of the DLPHSI, the system was set to a focal distance of 45 cm with the Spectralontarget filling the detector field of view (FOV), and a combination ofparameters based on the 2×2×2×2 factorial analysis model presented aboveare programmed into the GUI. Temporary data cubes are acquired to setthe exposure time that allows for a maximum response near 12,000 countsas seen in Table 4. Due to the illumination intensity of the “3 Shot”illuminations the exposure time could not be adequately lowered to keepthe detector from saturating under ‘high gain’ conditions.

TABLE 4 Exposure time (μs) required to get maximum response near 12,000counts for the given combination of parameters. Spectral Sweep 3 ShotRaw Oxy Raw Oxy 2 × 2 Low Gain 980 980 5750 5750 High 184000 184000saturates saturates Gain 4 × 4 Low Gain 200 200 1250 1250 High 4000040000 saturates saturates Gain

After the exposure times are determined, the system is set to acquirecontinuously for a period of ten (10) minutes. The number of outputbitmaps created within that ten (10) minutes is counted, and the time tooutput one processed image for the set parameters is calculated byequation 17.

$\begin{matrix}{t = \frac{600}{\#\mspace{14mu}{bitmaps}}} & (17)\end{matrix}$

Table 5 shows the results of all 12 runs. As expected from casualobservations, the fastest time to output one “3 Shot” image visualizingoxygenation is near ⅓ of a second (0.33 seconds), which means that underthese conditions, the system is operating a nearly 3 frames per second(fps). Acquiring Spectral Sweep images with 2×2 binning causes anexception violation in the software because the resulting image cube istoo large for MATLAB to process (696×520×126). Comparing Oxy to Raw, itappears that the processing algorithm has an increased effect on speedwhen binning is reduced, but no differential effect on speed when gainis increased. From the spectral sweep results, it also appears that theprocessing algorithm has a greater effect on speed than does gain level.These observations may be helpful in determining how to make imageacquisition and processing most efficient with the DLP HSI.

TABLE 5 Time (sec) to output one processed image for the givencombination of parameters. Spectral Sweep 3 shot Raw Oxy Raw Oxy 2 × 2Low Gain exception exception 0.48 0.71 High exception exceptionsaturates saturates Gain 4 × 4 Low Gain 14.63 23.08 0.32 0.39 High  9.6818.75 saturates saturates Gain

To further illustrate the timing of each process in the hyperspectralimage acquisition sequence, the timing diagram in FIG. 54 follows theflow of data from initializing acquisition to visualization of aprocessed bitmap. The closed loop describes acquisition of a singleslice of a hyperspectral cube, thus the time required to complete thatloop must be multiplied by the number of slices or illumination spectrain order to calculate acquisition time. By examining the time log of theGUI during acquisition, processing time for each algorithm is deduced.This processing time is subtracted from the output time in Table 5 for4×4 binning and low gain for “3 Shot” (high gain for spectral sweep) todetermine the total acquisition time. Dividing the total acquisitiontime by the number of slices equals the time to complete one cycle ofthe closed loop. For spectral sweep, the acquisition time is 116ms/slice, and for “3 Shot”, the acquisition time is 93 ms/slice. Thisdifference, 23 ms/slice, is less than the difference in exposure timefor the two methods, 38.75 ms/slice, indicating that initialization ofthe camera or light source plays a large factor in acquisition time.

FIGS. 55A and 55B are diagrams illustrating the processing algorithmsfor visualizing the images of blood oxygenation captured using thespectral sweep and the “3 shot” method of the present invention. FIG.55A shows the spectral sweep comparing measured spectrum to referencespectra via multivariate least squares analysis to quantify relativeconcentration HbO₂. FIG. 55B shows the “3 shot” method which subtractsthe image representing Hb absorbance from the image representing HbO₂absorbance and dividing the broadband absorbance to quantify relativeconcentration of HbO₂.

The methods are further described infra. To visualize the ischemiainduced by occluding blood flow to a finger and the ensuing reactivehyperemia upon removal of the occlusion, a subject's hand is imaged withthe DLP HSI in spectral sweep and “3 Shot” modes (FIGS. 56A and 56B,respectively). The system is warmed up, focused at a distance of 18 cm,and set to capture a background cube. The subject's hand was placed palmup in the FOV of the camera so that three fingers and part of the palmwere imaged. ‘Control’ hyperspectral image cubes, i.e., five spectralsweep cubes and five “3 Shot” cubes, were acquired of the non-occludedfingers. Each spectral sweep cube was analyzed with ‘Oxyz Jet’ and each“3 Shot” cube was analyzed with ‘3 shot jet (mid)17’ processingalgorithms, resulting in five output bitmap images color-coded forpercent HbO₂ for each illumination method. After acquiring ‘Control’images, a rubber band is wrapped three times around the base of thesubject's middle finger and five ‘Occluded’ hyperspectral image cubesfor each method were acquired and processed in the same way as the‘Control’ images.

To visualize the real-time progression of reactive hyperemia afterremoval of the occlusion, “3 Shot” images were acquired and processedcontinuously after acquiring the ‘Occluded’ images. While the systemcontinued outputting “3 Shot” images color-coded for percent HbO₂, therubber band was cut with a pair of scissors. After a period of threeminutes, the system was switched to spectral sweep mode to collect five‘Reperfusion’ hyperspectral image cubes.

The spectral sweep hyperspectral images were averaged for each timepoint in the experiment (n=5, for ‘Control’, ‘Occluded’, and‘Reperfusion’). The “3 Shot” output bitmaps are averaged for ‘Control’and ‘Occluded’ time points (n=5), and the mean pixel value for aconstant 49 pixel area represents percent HbO₂ at each time point. Forthe real-time progression of reactive hyperemia, the mean pixel valuefor the 49 pixel area was plotted versus time for several seconds priorto cutting the rubber band and 180 seconds immediately thereafter.

The results are now discussed infra. Visual inspection of the processedspectral sweep images (See FIG. 57A) showed that the DLP hyperspectralimaging system can differentiate between oxygenated and deoxygenatedtissue. In FIG. 57A, all three fingers were colored shades of red,orange, and yellow, which corresponded to a percent HbO₂ between 60 and80%, as indicated on the colorbar at the right. As expected under‘Control’ conditions, there was no difference in the surface oxygenationamongst the three fingers. In FIG. 57B, the middle finger was coloredblue-green, which corresponds to a percent HbO₂ between 40 and 50%,while the other two fingers appeared the same as they did in the‘Control’ image. As expected under ‘Occluded’ conditions, the rubberband tourniquet inhibits blood flow to the finger and effectivelyde-oxygenates the tissue. In FIG. 57C, the middle finger was coloredred-orange, corresponding to a percent HbO₂ higher than 75%. As expectedunder ‘Reperfusion’ conditions, the DLP HSI visualizes an overshoot intissue oxygenation related to reactive hyperemia most likely caused byvascular autoregulation of the previously occluded finger.

To quantify this overshoot related to reactive hyperemia, inspection ofthe 3 Shot output images was necessary. In FIG. 58A, the processed “3Shot” output image under ‘Occluded’ conditions matched its counterpartspectral sweep image (FIG. 57B). The non-occluded fingers appearedredder than those in the spectral sweep image, indicating a higherpercent HbO₂ than measured by the spectral sweep method. The two methodswere calibrated further to determine the truer method of measuringabsolute oxygenation. Because of this discrepancy in absolute measures,all results must be considered in relative terms rather than in absoluteoxygen percentages.

Inspecting the “3 Shot” images immediately after the occlusion isremoved and several minutes later (FIGS. 58B and 58C, respectively), itappeared that there is an immediate overshoot of percent HbO₂ in thepreviously occluded middle finger and a subsequent return to normaltissue oxygenation levels. This overshoot is quantified by calculatingthe mean of the pixel values in the black sample area of each imagedivided by the average ‘Control’ values for the same pixel area. Withthis calculation, the transient response of reactive hyperemia in themiddle finger was examined. In FIG. 59, the average percent HbO₂relative to control for the black sample area is plotted for every “3Shot” output image acquired between occlusion and reperfusion. Whileoccluded, the surface oxygenation was less than 70% of the controloxygenation, but in the first ten (10) seconds after removing thetourniquet the surface oxygenation raises to 112% of control.

Variability in consecutive images was apparent when viewing the outputimages in real-time and is also apparent in the wide vertical spread ofthe data points in FIG. 59. This image to image variability is notentirely understood, but may be an artifact of the OL 490 outputvariability. Even with the variability, the overall trend of oxygenationappeared to mimic an under-damped 2nd order system response. There is aninitial overshoot, then undershoot after about 1.5 minutes, with percentHbO₂ eventually returning to the control level after longer than 3minutes.

The DLP HSI can be used to visualize ischemia and reactive hyperemia, asevidenced by the proof-of-principle finger occlusion test. ProcessedSpectral Sweep output images showed high contrast between oxygenated andde-oxygenated tissue and indicated gross physiological changes.Processed “3 Shot” output images showed the same contrast betweenoxygenated and de-oxygenated tissue, but the two illumination andprocessing methods do not give identical absolute measurements ofpercent HbO₂.

Images acquired and processed by the “3 Shot” method were generated atabout 3 frames per second, while spectral sweep images are generated atabout 3 frames per minute. Both are useful methods for visualizing thespatial distribution of surface tissue oxygenation, but the “3 Shot”method is preferable when visualizing short duration physiologicalchanges. Analyzing several minutes of continuous “3 Shot” outputs showsreactive hyperemia of the surface tissue in a previously occluded middlefinger.

The following descriptions detail particular examples of procedures,conditions, etc. which are applicable to utilizing the present invention“3 Shot” methods.

Partial nephrectromy is a surgical procedure in which a section of apatient's kidney is removed, usually to eliminate a tumor. Sparing theremainder of the kidney often requires vascular occlusion in order totemporarily interrupt renal blood and prevent hemorrhage. Two methods ofrenal vascular occlusion: artery-only occlusion (AO) and artery and veinocclusion (AV), have demonstrated differential effects on renal tissueviability during 2 to 24 hour long periods of ischemia. (See Gong, E M,Zorn, K C, Orvieto, M A, Lucioni, A, Msezane, L P, Shalhav, A L,Artery-only occlusion may provide superior renal preservation duringlaparoscopic partial nephrectomy. Urology 2008; 72:843-846). Tounderstand the effects of competing clamping methods during shorterperiods of ischemia, the DLP HSI was used to image the renal tissueoxygenation during partial nephrectomies in pigs and then humans forperiods of ischemia less than one hour long.

The following describes the results of the study involving pigs.Following approval by the UTSW Institutional Animal Care and UseCommittee (IACUC), four female Yorkshire pigs, weighing between 60-80kg, underwent AO or AV occlusion of each kidney. At the beginning of thestudy, one kidney was used to perfect the investigational technique and,therefore, was not included in the analysis. In order to verify theimages obtained by the prototype DLP HSI, hyperspectral image cubes areacquired by sweeping 126 contiguous bandpasses, identical to thespectral sweep illumination method, with an already characterizedLCTF-based hyperspectral imaging system. The primary difference betweenthe two systems is the method of filtering light. In the LCTF systembroadband light is reflected from the tissue sample and then filtered bya liquid crystal tunable filter (LCTF) before being detected by aCoolSNAP HQ2 array detector. In the DLP system's spectral sweep method,narrowband light from the spectral light engine is reflected from thetissue sample and directly detected by a CoolSNAP HQ2 array detector.There is no LCTF equivalent to the DLP “3 Shot” method as a LCTF isincapable of providing complex spectra as provided by the DLP.

After anesthetizing each subject according to protocol, the surgeonapproached the right kidney through an open midline incision and pulledthe peritoneum away from the visible surface of the kidney. While thesurgeon exposed the kidney, background cubes for both illuminationmethods are acquired in a dark corner of the surgical suite. The DLP HSItripod was then abutted to the surgical bedside (See FIG. 60) and thefriction head was adjusted so that the exposed kidney is fully visiblein the camera's FOV. The wheels of the tripod dolly were locked in placeso that all images are acquired from the same angle and distance.‘Control’ hyperspectral image cubes are acquired and processed byspectral sweep and “3 Shot” methods as in the finger occlusion testdescribed above. The surgeon then dissected the hilem and clamped eitherthe artery only (AO) or the artery and the vein (AV) with a curvedSatinsky clamp.

The renal vasculature remained occluded for one hour before removing theclamp. Spectral sweep ‘Occluded’ image cubes are acquired and processedin approximately 5 minute intervals throughout the occlusion. “3 Shot”image cubes were acquired and processed continuously during applicationof the clamp, several times during the hour of occlusion, andcontinuously during removal of the clamp. After one minute ofreperfusion, spectral sweep ‘Reperfusion’ image cubes were acquired at 5minute intervals for 30 minutes. Following ischemia and reperfusion ofthe right kidney, the identical procedure was carried out on the leftkidney using the opposite clamping technique. During the procedure, thesubject's rectal temperature was maintained between 37° C. and 39° C.and its serum oxygenation, as measured by a pulse oximeter on theanimal's ear, was kept between 98% and 100%.

Percent HbO₂ was measured over a uniform 81 pixel area chosen from thecenter of each kidney at each time point during the study. An average ofthe pixels for each region was calculated along with a standarddeviation for each image cube. Images from each 10 minute time frameduring ischemia were grouped as a single dataset in order to determinedifferences between AV and AO clamping conditions at each of six timeperiods. A regression model was fit to the individual levels using SASstatistical software (SAS Institute, Cary, N.C.) in order to determinewhether there is a statistically significant difference between clampingmethods with regard to kidney tissue ischemia over a period of one hour.

Real-time progression of ischemia and reactive hyperemia was measured ina 225 pixel area for all of the “3 Shot” output bitmaps acquired duringtightening and removal of the Satinsky clamp in one pig subject. Themean pixel value in each image's sample area was plotted as a functionof the time the output image was originally processed. This plot showedthe transient behavior of the initial decline of percent HbO₂ afterocclusion and the extent of reactive hyperemia after the occlusion isremoved. It is unknown whether these immediate transient effects will bedifferent between AV and AO clamping.

In the porcine kidney studies, “3 Shot” images appear similar tospectral sweep images (See FIG. 61). Comparing FIGS. 61C and 61D toFIGS. 61E and 61F, the relative tissue oxygenation is the same for bothillumination methods. It is obvious that between ‘Control’ and‘Occluded’ states there is a drop in surface oxygenation for the renalunit, but the surrounding muscle and adipose tissue remains highlyoxygenated. The major difference between spectral sweep and “3 Shot”processed images is the threshold vector by which the final image ismapped to percent HbO₂. For the spectral sweep method, the spectralinformation stored in the 126 slice data cube allows a wider range ofvalues, so the background is truly blue, representing virtually nooxygenation. For the “3 Shot” method, the range of pixel values in aprocessed image is much less than that for a spectral sweep image, so atighter threshold must be applied to visualize the same contrast. Thistighter threshold causes the mapped percent HbO₂ values to be slightlydifferent than those mapped in the spectral sweep images, so thedetectable range and resolution of percent HbO₂ may be less with the “3Shot” method than with the spectral sweep method.

The overall trend of pixel intensities in “3 Shot” output images isidentical to the trend of spectral sweep images and matches the expectedoxygenation trend. Therefore, “3 Shot” output images are sufficient formapping the spatial distribution of tissue oxygenation over time. Forthe first three pig subjects, continuous acquisition of “3 Shot” imagesduring application and removal of the Satinsky clamp was not performedbecause the primary concern was capturing quality images to test hourlong effects of clamping. Before imaging the fourth pig subject, datafrom the other subjects indicated that a majority of tissue chemistrychanges occur during the first minute after clamping and during thefirst few minutes after removing the clamp. So, for the fourth pigsubject, continuous “3 Shot” images are acquired during both criticaltime periods for each kidney.

A 225 pixel area from the center of the kidney is averaged in each imageand divided by the average of the same area for the first 20 controlimages to result in the percent HbO₂ relative to control. In FIG. 62,this value is plotted for the kidney clamped AO (n=247) and for thekidney clamped AV (n=217) for 12 seconds prior to tightening of theclamp and about one minute following application of the clamp. For thefirst 10 seconds of occlusion both methods of clamping result in alinear decline of oxygenation. The transient response then becomesnon-linear, and flattens after 30 seconds in both cases. There is azero-order time delay between the two responses, with the AO declinelagging behind the AV decline. This lag may be due to the flow of bloodthrough the renal vein, which would not occur under AV clamping. Thereis also a steady-state difference between the oxygenation levels, withAO being higher than AV.

The “3 Shot” outputs acquired during the removal of the Satinsky clampare analyzed using the same procedure for the AO clamped kidney (n=1540)and the AV clamped kidney (n=269). Images are continuously acquireduntil it visually appears that there is no longer any fluctuation inkidney oxygenation levels. From the number of images in each case, it isapparent that visual fluctuations cease more quickly in the AV case thanin the AO case. Another interesting observation is that the AO case hasan overshoot of oxygenation associated with reactive hyperemia and thendeclines to a steady state lower than its control value, and lower thanthe AV steady state. This may be a potential detriment to clamping theartery only instead of the artery and the vein. Perhaps, there is alonger term oscillation that is not captured due to lack of images inthe AV case.

An unanticipated advantage of near real-time “3 Shot” images showing thesurgeon the spatial distribution of tissue oxygenation is illustratedwhen clamping fails to cause the entire kidney to become ischemic. Inpig 3, the surgeon clamped the renal vasculature AO as in previous pigsubjects. Raw views of the kidney seem to imply the clamping issuccessful (See FIG. 63A), but as the surgeon watches the real-time “3Shot” output display in the GUI, he notices that the upper pole is stillhighly oxygenated but the lower pole has become ischemic (See FIG. 63B).This prompts a re-examination of the renal vasculature, in which thesurgeon finds a second artery supplying oxygenated blood to the renalunit. Apparently, in this case, the clamped artery supplies blood to thelower pole, and an unclamped second artery supplies blood to the upperpole. Upon clamping the second artery, the upper pole also becomesischemic, but never reaches the same level of ischemia as the initiallyoccluded lower pole (See FIG. 63C). Therefore, the DLP HSI can also beused to determine the precise tissue areas that are supplied by anartery, which helps surgeons to know which arteries to occlude duringpartial nephrectomies.

Hyperspectral imaging with the DLP HSI is useful for comparing competingrenal vasculature clamping methods with regard to kidney tissueoxygenation during nephron-sparing surgery. Analyzing spectral sweepimages indicates that the percent HbO₂ of the kidney is higher when theartery only (AO) is clamped than when the artery and vein (AV) isclamped between 10 and 40 minutes. Images from other time periodsindicate no difference in percent HbO₂ between AO and AV clamping. Thespectral sweep image cubes captured by the prototype system visualizethe same spatial oxygenation map as other hyperspectral imaging systems,even though the spectrum measured at each pixel is not always consistentbetween systems.

Generating “3 Shot” output images at 3 frames per second allows thesurgeon to visualize real-time physiologic changes in tissueoxygenation. This is useful in determining whether or not blood flow isadequately inhibited from the entire kidney. Analyzing “3 Shot” imagescaptured during the process of tightening and releasing the clamp showsthe transient response of renal tissue at the onset of ischemia andduring reactive hyperemia. Though only one subject was sufficientlyimaged during this time period, it appears that AV clamping may beadvantageous to AO clamping when immediate ischemia or no hyperfusion isdesired.

The following discussion is related to similar studies performed onhuman subjects. Animal studies are necessary to ensure the safety andutility of prototype medical devices, but the end use of the DLP HSI isfor human imaging, so its utility in live human surgeries must also betested. An Institutional Review Board (IRB) approved protocol at UTSWpermits the prototype hyperspectral imaging system to image open cavitypartial nephrectomies in human subjects.

The following methods were used in the human studies. The surgicalprocedure for humans is different than that for the porcine study,because human subject survival is of the utmost importance. The surgeonmakes an incision in the side of the subject to expose the kidney.‘Control’ hyperspectral data cubes are acquired with the DLP HSI inspectral sweep and “3 Shot” modes. Then, the surgeon clamps the renalvasculature with a predetermined method (AO or AV) to occlude blood flowto the kidney. ‘Occluded’ hyperspectral data cubes are acquired atnon-critical points in the surgery, when the surgeon allows all otherroom lights to be turned off. After the kidney is sufficiently bereft ofblood to prevent hemorrhage, the surgeon removes the tumor and stitchesthe exposed renal tissue closed. One more ‘Occluded’ data cube isacquired before the clamp was removed. The surgeon removed the clamp,and ‘Reperfusion’ hyperspectral data cubes are acquired prior to closingthe abdominal cavity.

To date, three human subjects have been imaged. The following resultssection is focused on the first of these human subjects. The subject isan otherwise healthy male, about 60 years old, with a tumor on his rightkidney.

In human surgery, the room lights and operating lights cannot beswitched off for minutes at a time as is possible during animalsurgeries. Therefore, only one spectral sweep data cube was captured asa control to verify the image results of the “3 Shot” illuminationmethod. Subsequent hyperspectral images were captured with the “3 Shot”method so that the room and operating lights only remain off for severalseconds.

View finder pictures of the kidney show the gross anatomy before andafter the tumor is removed, and “3 Shot” image outputs show the relativeoxygenation of the kidney throughout the surgery. In FIG. 64A, the bluesample area highlights a uniform region of the kidney tissue. The samplearea is used to trace the trend of oxygenation throughout the surgery.

Hyperspectral images are acquired at five different time points duringthe surgery. One of the images is unreliable due to light pollution fromthe headlamp worn by the surgeon, and the other four images are shown inFIGS. 64B-64E. In the ‘Control’ image, FIG. 64B, the kidney is highlyoxygenated and the spatial distribution of tissue oxygenation is fairlyuniform. In the first ‘Occluded’ image, FIG. 64C, most of the kidneyexhibits lower oxygenation levels. After the tumor is removed and theclamp is still tightened, FIG. 64D, the kidney shows decreasedoxygenation levels in all areas. When the clamp is finally removed, FIG.64E, the kidney oxygenation levels return to near those measured in the‘Control’ image.

In FIG. 65, the average and standard deviation of the 81 pixel values ineach sample area are plotted to show the trend of oxygenation throughoutthe surgery. The overall trend implies that the kidney becomes mostischemic in image three and then recovers to a level similar to control.

The DLP HSI can successfully map the spatial distribution of tissueoxygenation during a human partial nephrectomy. Light pollution from theoperating room lights and the surgeons head lamp interferes withhyperspectral imaging, so fewer images can be captured in human casesthan in animal cases. However, the “3 Shot” output images acquiredduring the human study are useful for the surgeon to visualize whichareas of the kidney are higher and lower oxygenated. Analysis of manymore subjects is needed in order to compare AO versus AV clamping inhuman nephron-sparing surgeries.

Nerve damage may be correlated to the loss of blood flow in thesurrounding tissue bed. To diagnose nerve damage in the lower limb ofpatients, physicians probe the leg with a mechanical or electrical prickand ask the patient if they can feel it. If the patient cannot feel theprick, that area is diagnosed as neuropathic. This process is timeconsuming, and the spatial resolution of the diagnosed neuropathy is afunction of the number of discrete point measurements made by thephysician. With the DLP hyperspectral imaging system it may be possibleto map the spatial distribution of neuropathy along the entire lowerlimb by mapping its tissue oxygenation.

The following discussion is related to measurements of lower limbneuropathy. A clinical study is devised to test the utility ofhyperspectral imaging for monitoring wound care in amputees and forstudying their neuropathy. Amputees and patients exhibiting lower limbneuropathy are voluntarily enrolled in the clinical study at a weeklywound care clinic in the Dallas Va. After signing the consent forms,each subject is seated in a room reserved for hyperspectral imaging, theoverhead lights are turned off, and the DLP HSI is used to capturespectral sweep and “3 Shot” images along the entire limb of interest.

To date, five human subjects have been enrolled in the study. Thefollowing results section is focused on the first of these humansubjects. The subject is a male over 50 years old who has previouslybeen diagnosed with neuropathy in the lower half of his left leg, asmeasured by the prick test.

Spectral sweep and “3 Shot” hyperspectral image cubes are acquired fromthe subject's toes to his knee in the lower limb's frontal plane. Theoutput images color-coded for percent HbO₂ with the “3 Shot” methodshows similar relative oxygenation levels to the color-coded outputs ofthe spectral sweep method. Overlaying the spectral sweep images on apicture of the subject's leg creates a surface tissue oxygenation mapfor the entire leg (See FIG. 66).

Due to the curvature of the ankle, the detected absorbance imagescaptured between the foot and the shin contain an unreal intensitygradient and are not shown. Those images unsuccessfully predicted thespatial map of tissue oxygenation, emphasizing an important constraintof the prototype: the topography of the tissue being imaged. Tissueplanes further from the camera and light source optics may appear lessoxygenated than closer tissue planes, when their real oxygenation is thesame.

It is difficult to see any difference in oxygenation from the bottomhalf of the leg to the top half of the leg in the images alone. A samplearea of pixel values along the midline of the leg is plotted versus theaxial distance from the toes (See FIG. 67) to quantify this difference.This plot indicates that there is an increase in surface tissueoxygenation progressing from the toes to the knee. However, thisincrease does not seem to match the results of the prick test. Ifsurface tissue oxygenation mimics underlying neuropathy, there should bea jump in oxygenation between the region of neuropathy and the normalregion, but the hyperspectral images do not show this jump. Thisemphasizes another important constraint of the prototype: the depth ofthe tissue being imaged. The DLP HSI relies on measuring the absorbanceof light in the visible wavelength range, and visible light does notpenetrate deep into the tissue, so the measured percent HbO₂ is onlyindicative of epithelial tissue oxygenation. Epithelial tissue is highlyregulated and its oxygenation levels do not correspond to neuropathylike deeper tissue oxygenation does. Development of a prototype thatoperates in the near infrared wavelength range may be useful forvisualizing deeper tissue oxygenation.

The DLP HSI is useful for mapping the spatial distribution of tissueoxygenation for areas that are larger than the field of view of thecamera. Several image cubes are captured in overlapping regions, and theresulting output images are overlain to create a large spatial map.

When acquiring images with the DLP HSI, it is important to know thetopography and depth of the tissue of interest. The tissue area shouldbe a uniform distance from the system's detector and light sourceillumination optics to eliminate unreal effects of distance on measuredpercent HbO₂. The tissue of interest should also be on the surface ofthe subject (e.g., epithelium), since visible light does not penetratepast the surface layers of tissue. In the case presented, epithelialoxygenation does not seem to correlate exactly with neuropathy, but alarger sample size is needed to confirm these results.

The following discussion is related to measurements taken during brainsurgery. Neurosurgeons investigate brain behavior by dissecting thescalp and viewing the cerebral cortex with surgical microscopes. Currentimaging only visualizes the physical anatomy of the cortex, or byintroducing dyes as contrast agents some functionality can bevisualized. Point measures of cerebral HbO₂ content are possible withnear infrared spectroscopic techniques, but they do not show a spatialimage of oxygenation. (See Quaresima, V, Sacco, S, Totaro, R, Ferrari,M, Noninvasive measurement of cerebral hemoglobin oxygen saturationusing two near infrared spectroscopy approaches. 2000; 5:201-205.) Bycoupling the DLP hyperspectral imaging system to a surgical microscope,neurosurgeons may be able to visualize the spatial profile of cerebraltissue chemistry.

The CoolSNAP HQ2 is directly mounted to the C-mount of a Zeiss surgicalmicroscope at the UTSW animal laboratory, and the OL 490 liquid lightguide is coupled to the microscope's ring illumination optics. Aftersetting the focal distance of the microscopic system, a background cubeis captured with the DLP HSI and the surgeon dissects a smallcross-section of an anesthetized rabbit's scalp. Imaging through theoptics of the microscope, the DLP HSI is set to continuously acquire “3Shot” images of the exposed cortical tissue. Several seconds of controlimages are captured while an external pump supplies oxygen to thesubject. Continuous “3 Shot” images are acquired while the externaloxygen supply for the subject is cut off for a period of 20 minutes andturned back on.

Watching the spatial distribution of tissue oxygenation, and drawingfrom his own experience, the surgeon notices one area of the exposedcortex is damaged at the beginning of the imaging session. In FIG. 68A,the central region is the cortical tissue and the three red regionsaround the circumference are scalp and other tissues. The black samplebox surrounds an area of healthy brain tissue and the blue sample boxsurrounds an area of damaged brain tissue. In the control image, thedamaged tissue appears ischemic compared to the normal tissue's level ofoxygenation. There are no noticeable changes in the damaged tissueoxygen levels shown in the hyperspectral images during the period ofexternal oxygen cutoff. However, after the oxygen is cut off for severalminutes, the normal brain tissue begins to show ischemia as indicated bythe lower percent HbO₂ in the black sample area of FIG. 68B.

Initial brain imaging with the DLP HSI coupled to a Zeiss surgicalmicroscope suggests neurosurgery is another viable application for thisnovel medical imaging platform. The digital visualizations ofoxygenation generated by hyperspectral imaging correspond with what thesurgeon expects to see. Even though conclusive medical results cannot bederived from the initial imaging session, the results indicate that DLPHSI can enable surgeons to visualize cerebral tissue chemistry in nearreal-time.

In view of the foregoing description of the present invention, it shouldbe appreciated that a variety of procedures and diagnostic techniquesmay be improved by incorporation of the present invention. The followingdiscussion describes some of such procedures and diagnostics techniques.This discussion is not intended to limit the scope of the invention asset forth in the appended set of claims.

There are a variety of applications of hyperspectral imaging of theretina. For example, measuring oxygenation of the retina as may beeffected by diabetic retinopathy, central retinal vein occlusion, branchretinal vein occlusion, central retinal artery occlusion, branch retinalartery occlusion, sickle cell retinopathy, retinopathy of prematurityand retinal vascular inflammatory diseases. Diabetic retinopathy is themost common cause of new cases of blindness in adults 20-74 years ofage. Retinal vascular occlusions, sickle cell retinopathy andretinopathy of prematurity can also lead to irreversible severe visualloss. These conditions often lead to retinal ischemia and subsequentretinal neovascularization with the associated blinding complications ofretinal detachment, vitreous hemorrhage and neovascular glaucoma. Laserphotocoagulation of the peripheral retina can markedly reduce the riskof these complications. Targeting the areas of retina that are sufferingfrom ischemia improves the clinical outcomes while reducing peripheralvisual field loss. Furthermore, these conditions can also lead tomacular edema (i.e., swelling of the central retina) and macularischemia. Distinguishing between these two causes of visual loss helpsto target patients that would benefit from light macular laser therapyversus pharmacologic interventions. Traditionally, fluoresceinangiography (FA) has been used to evaluate for both peripheral retinalischemia and macular ischemia. However, FA is painful, has risksassociated with intravenous injections, leads to discoloration of theskin for twenty four (24) hours, can lead to allergic reactions of alldegrees of severity, often leads to nausea and vomiting, and may beimpossible to perform in patients with poor vein access. Furthermore,the technique evaluates vascular flow and permeability rather thanretinal oxygenation. It is believed that hyperspectral imaging of theretina measuring oxyhemoglobin and deoxyhemoglobin may allow for earlieridentification of areas of ischemia before permanent loss of capillaryperfusion has occurred. It also would allow for a non-invasivedetermination of retinal oxygenation. This would speed up and simplifythe process of obtaining this clinically important piece of information,while reducing the risk of complications and eliminating the associatedpain and discomfort.

The present invention hyperspectral imaging system may also be used formeasuring optic nerve oxygenation. Such information may be useful inevaluating such conditions as glaucoma, ischemic optic neuropathy andoptic neuritis. It is believed that assessment of optic nerve headoxygenation may help diagnose patients with increased risk for glaucomaor ischemic optic neuropathy. It may also help differentiate betweenpatients with ischemic optic neuropathy versus optic neuritis.

The present invention may also be used for measuring macular pigmentsthereby providing information regarding age-related macular degenerationand juxtafoveal telangiectasis. Age-related macular degeneration (AMD)is the most common cause of irreversible blindness in patients oversixty (60) years of age. Despite advances in therapeutics, clinicaloutcomes are still suboptimal. There is no effective therapy foratrophic AMD. Macular pigments including lutein and zeaxanthin arepigments that are present in the retina in the macular region. Theyabsorb particularly harmful wavelengths of light (i.e., blue light) andprotect the retina from phototoxicity. These pigments have been shown todecrease with age and more rapidly in patients with AMD and juxtafovealtelangiectasis. It is believed that the present invention provides aneasy modality to measure the levels of macular pigments which in turnmay help to identify patients that would benefit from dietarysupplements with lutein before they develop clinical disease.

The present invention may also be used for measuring pigments in theretinal photoreceptors and retinal pigment epithelium thereby providinginformation related to retinitis pigmentosa and other hereditary retinaldegenerations, e.g., age-related macular degeneration. Hereditaryretinal degenerations and age-related macular degeneration lead toabnormalities in the photoreceptors and underlying retinal pigmentepithelium (RPE). The health of the photoreceptors and the retinalpigment epithelium can be correlated to the levels of pigments (i.e.,rhodopsin, lipofuscin and melanin). It is believed that the presentinvention will permit measuring these pigments accurately therebyallowing better monitoring of the clinical course in these patients.Moreover, it is believed that the present invention will also allowmeasurement of the response to prophylactic interventions. Finally, itis believed that the present invention may aid in the early diagnosis ofthese conditions or subclassification into clinically relevant subgroupsdue to its ability to obtain the foregoing measurements.

The present invention may also be used for the diagnosis of autoimmuneretinitis, infectious retinitis and infiltrative neoplastic conditions.There are a limited number of physical changes that the retina canundergo in response to injury. The clinical presentation of retinalautoimmune infiltration (e.g., Wegener's granulomatosis, sarcoidosis,etc.), infectious retinitis (e.g., cytomegalovirus, toxoplasmosis andherpetic retinitis as in acute retinal necrosis or progressive outerretinal necrosis), and neoplastic infiltration (e.g., lymphoma andleukemia) can sometimes be indistinguishable from one another. In thosecases, surgical intervention to obtain vitreous and/or retinal tissuefor diagnostic purposes is often necessary. There are significant risksassociated with this procedure. However, lack of therapy would oftenlead to blindness or loss of the eye. Furthermore, an erroneousdiagnosis leading to an erroneous therapeutic intervention canprecipitate acute exacerbation of the disease (e.g., treating a viralinfection with steroids) or delayed diagnosis (e.g., anti-inflammatorytherapy for a neoplastic condition) resulting in loss of vision, loss ofthe eye or progression of a life-threatening disease. It is believedthat the present invention hyperspectral imaging system and methods canlook for spectral signatures of pathogens thereby helping to ascertain adiagnosis in a non-invasive and rapid way.

The present invention may also be used for non-invasive evaluation ofdisease biomarkers. It has been said that the eye is a window to thebody. The eye provides the clinician with an unobstructed view ofvessels. This has for years allowed medical professionals to diagnosesystemic conditions like diabetic and hypertensive retinopathy. It isbelieved that hyperspectral imaging will allow medical professionals totap deeper into this invaluable resource. Not only can the spectrum ofsubstances in the visible light spectrum be identified likeoxyhemoglobin and deoxyhemoglobin. The spectral signatures of a widerange of biomarkers both in the visible and invisible light spectrum canalso be identified. This will allow medical professionals to diagnoseand non-invasively monitor a wide range of conditions, including butlimited to depth of anesthesia during surgical cases, systemicintoxications, drug overdose, pancreatitis, liver disease, prostate andother cancers with known biomarkers, metabolic diseases, infectiousdiseases with pathogens that produce specific spectral signatures, etc.Furthermore, it is believed that it will allow medical professionals todiscover new spectral signatures that correlate with common and uncommondiseases and that could serve as new biomarkers in the diagnosis andmanagement of those conditions.

Moreover, the present invention may also be used to monitor the clinicalcourse of battlefield blast injuries based on retinal vasculaturediameter changes. It should be appreciated that the foregoingdescription of opthalmological applications are but some of theprocedures and diagnostic methods that can be improved by theincorporation of the present invention hyperspectral imaging system andmethods. For example, as discussed supra, real-time diagnosticmeasurements of the kidneys provides surgeons with never beforeavailable information to use during surgery.

Surgery is the mainstay treatment for patients diagnosed with kidneycancer. As such, removal of the tumor by either partial or totalnephrectomy will inherently decrease renal function and risk thedevelopment or progression of chronic kidney disease. Chronic kidneydisease (CKD) is prevalent in 11% of the adult population, including 17%of patients older than 60 years. It is associated with an increased riskof hospitalization, cardiovascular event, and mortality from any cause.In fact, patients with moderate kidney disease (CKD, stage 3-GFR<60ml/min/1.73 m²) have a 40% increased risk of cardiovascular events and20% increased risk of death. As such, for patients who have beendiagnosed with kidney cancer, most of which are over 60 years old andrequire surgery, preserving as much renal function as possible is thestandard of care and may prevent cardiovascular morbidity and mortality.

As approximately 80% of renal tumors identified are small (i.e., StageTi which are typically <7 cm), kidney sparing surgery (e.g., partialnephrectomy) is the preferred technique to maximally preservefunctioning kidney tissue. However, to comfortably and safely performthis procedure, the urologic surgeon routinely clamps the renal arteryto stop blood flow to the kidney (i.e., ischemia). This improvesvisualization and minimizes blood loss. However, the resulting ischemiarisks kidney injury so that surgeons often will cover the kidney withice slush for 7-10 minutes prior to tumor excision in an effort todecrease kidney metabolism which is highly oxygen dependent (i.e.,aerobic metabolism). Nevertheless, warm ischemia (i.e., no ice) of morethan 20 minutes, and cold ischemia of more than 30 minutes can result inirreversible damage in the remaining kidney. Therefore, despite the goodintent of preserving as much kidney as possible during cancer surgery,the patient and surgeon still risk decreased kidney function secondaryto the surgical technique.

Thus, if partial nephrectomy can be performed with only partialocclusion of the renal artery such that the kidney still gets oxygen thekidney is less likely to suffer from irreversible kidney damage.Unfortunately, there is no guideline as to what degree should the arterybe occluded and, more importantly, what is the minimum oxygen necessaryto protect the kidney. This is critical information since incompleteartery occlusion could increase blood loss and diminish visualizationduring surgery. Although the ability to monitor tissue oxygen levels inreal-time is now available using tissue probes that are often used inneurosurgery, the present invention hyperspectral imaging system andmethods provides a viable and effective alternative to these probes.Additionally, the present invention, unlike the probes, provides suchinformation in a non-invasive manner. As described above in view of theporcine and human studies, using the present invention which includesadvanced camera technology and novel software, light reflected off thesurface of an imaged tissue or organ can be used to measure oxygenlevels. This is an improvement over existing tissue oxygen monitorswhich must use a needle probe and can only measure oxygen at onelocation. Incorporation of oxygenation and blood flow monitoringtechnology in partial nephrectomy can warn surgeons when oxygen levelsare dangerously low and perhaps guide them in how long and to whatdegree the renal artery is partially or completely clamped. Recentlaboratory data gathered using an animal model has demonstrated that 50%occlusion of the renal artery maintained kidney oxygen levels longer andat higher values during partial nephrectomy. Though blood loss wasincreased slightly, it did not increase operative time or surgerydifficulty. Most importantly, when compared to conventional partialnephrectomy, the cases where tissue oxygen levels were higher (e.g., 50%artery occlusion) had less immediate and sustained kidney dysfunctionafter surgery. In view of the foregoing, it is believed and shown inpart above that the present invention hyperspectral imaging system andmethods can be used to monitor the vascular blood flow and monitortissue oxygen in clinical use and determine if kidney function can beprotected and improved when performing partial nephrectomy withreal-time monitoring of kidney oxygen levels.

The present invention will thus ensure that the recovery of kidneyfunction after partial nephrectomy will be improved by real-timemonitoring and minimization of the ischemic insult routinely incurred.This present invention has the potential to alter the way kidney surgeryfor cancer is performed. The preservation of renal function during andafter partial nephrectomy would be incrementally improved benefitingmany patients with compromised kidney function.

The present invention may also be used to monitor, analyze and diagnoseblast brain injury (BBI) and traumatic brain injuries (TBI).Overpressure brain injury is one of the most common injuries experiencedby soldiers. Soldiers are exposed to various levels of blast pressureranging from relatively low pressure that occurs from firing of weaponssystems to very high pressures that occur from explosions of IEDs andmortar rounds. Damaging pressure waves are magnified by reflection fromadjacent structures or the ground resulting in exposure of human tissuesto extremely high pressure. Pressure can also be transmitted to thebrain via the thorax when body armor is struck by high velocityprojectiles.

Blast pressures can cause neural injury by several mechanisms. Pressurewaves can cause substantial pulmonary injury with alveolar disruption.High pressure air injected into the blood can cause air emboli resultingin brain ischemia. Brain tissues can be injured directly from pressurewaves passing through the cranial vault. High pressure can also betransmitted to the brain through the blood when high pressure occurs inthe thoracic cavity. Lastly, brain function may be depressed bycirculating metabolites and cytokines that are released into the bloodstream by non-neural tissues that are injured by the blast wave.

Monitoring the severity and progression of blast-induced brain injury iscritical to ensuring optimal care for wounded soldiers. The presentinvention can be used in the battlefield to objectively assess thedegree of these injuries to facilitate appropriate triage of soldiersexposed to explosions. The present invention can scan the retina todetect appropriate biomarkers associated with the brain injury anddetermine the extent of injury.

The present invention may take one of several embodiments for performingthe foregoing analysis. A direct fundoscope will enable imaging of theretina to examine for retinal hemorrhage. This system may be augmentedto wirelessly connect to trained personnel in any location in the worldwho can interpret these images. Alternatively, hyperspectralmeasurements of retinal oxygenation may be taken with the presentinvention. It is known that retinal hyperemia is associated with blastinjury and therefore overall and regional differences in retinaloxygenation can be measured. Moreover, measurement of air emboli may betaken with the present invention. Air emboli have been implicated as acause for blast injury associated brain injury. Using the presentinvention, air bubbles in the retinal arterial blood supply may beidentified and quantified. Further, it is believed that hyperspectralimages of corneal surface blood vessels provide another means foranalyzing traumatic brain injury.

Lastly, the present invention may be used to measure blood chemistry.Blast injuries cause widespread tissue damage resulting in elevatedserum levels of leukotrienes C4, D4 and E4 and of 6-keto-PGF1 alpha andTxB2, the stable products of prostacyclin (PGI2) and thromboxane A2(TxA2). These substances are elevated to a much greater extent in blastinjury than blunt head trauma suggesting that these and other chemicalsmay serve as specific markers for the presence, extent and progress ofblast injury to the brain. The eye can serve as a window to themicrocirculation. Using visible spectroscopy, chemicals in the bloodwill be detected and quantified in the retinal microcirculation. It isbelieved that the chemical environment of the retinal microcirculationcorrelates to the magnitude of brain dysfunction experienced after blastinjury. This complex chemical environment will be manifested as complexspectra that will be unique for the blood chemical environmentassociated with various stages of blast injury. It is further believedthat the statistical classification of these spectra will enable thediagnosis and characterization of various stages of blast injury.

Further diagnosis may be performed by a near infrared version of thepresent invention. Such a system will enable chemometric assessment oftissues as deep as 1 centimeter and provide more informative spectralinformation than exists in the visible light range. This system willenable imaging of the retina and may provide information about bloodflow changes in the face that occur as a result of blast injury. In afurther embodiment, the present invention may include an infraredimaging system operating in the thermal range (5-15 micron) that willfacilitate tissue spectroscopy in the wavelength range where molecularvibrations attenuate infrared energy. Attenuation of black bodyradiation by molecular vibration will enable development of scanningdevices that will detect blood and tissue chemical composition with ahigh degree of discrimination for various disease states. It is believedthat minor changes in blood chemistry associated with various diseasestates will result in highly unique spectral signatures in this regionof the electromagnetic spectrum.

The foregoing measurements may be obtained by combining a conventionalfundus camera and the present invention hyperspectral imagingillumination system.

A further embodiment of the present invention provides a convenientmeans for threat detection. The present invention hyperspectral imagingsystem provides for the detection of pathogens and poisons on surfacesof equipment, assembly lines, etc. For example, the present inventionmay be used to screen a food supply for toxic agents, e.g., salmonella(which has a known strain-specific IR spectra), anthrax, botulina, smallpox, E. coli, shigella, etc. Further the present invention may be usedto detect various chemicals, such as, organophosphates pesticides, ricintoxin, sarin, soman, tabun or VX gas.

A novel hyperspectral imaging system has been developed for real-time,non-invasive, in vivo imaging of tissue chemistry in surgical andclinical applications.

Using a spectral light engine to illuminate with simple narrowbandpasses or complex broadband spectra and a CCD FPA to detect thereflectance image from a tissue sample, the DLP hyperspectral imagingsystem remotely visualizes tissue oxygenation in live human and animalsubjects. Illuminating with many narrow bandpasses in the spectral sweepmode results in large data cubes with full spectral information for eachpixel in the image, but requires 20 to 30 seconds to generate a singlechemically-encoded image. Spectral sweep data cubes acquired with theDLP HSI match those captured by LCTF-based hyperspectral imagers, which,along with color tile calibration results, validates the DLP HSIprototype as an imaging spectrophotometer. Illuminating with only threecomplex broadband illuminations in the present invention “3 Shot” moderesults in small data cubes with adequate spectral information togenerate chemically-encoded images identical to the spectral sweepimages at a much faster rate, i.e., 3 processed images per second.Typically, the wavelengths to be used depend on the tissue type due tothe penetration of tissue depth. For example, in visible spectra, HbO₂,Hb, HbCO₂ can be measured. In near-infrared (NIR) spectra, HbO₂, Hb,water and lipids can be measured. As shown in FIGS. 69A through 69C, thepresent invention can provide chemically encoded images. FIG. 69A showsan image of a human hand which has experienced a burn. FIG. 69B shows ahyperspectral image of the hand from FIG. 69A displaying chemicallyencoded data for relative contribution of oxyhemoglobin in the hand.Contrarily, FIG. 69C shows a hyperspectral image of the hand from FIG.69A displaying chemically encoded data for relative contribution ofwater in the hand. Thus, it can be seen from these figures that thepresent invention is capable of displaying a variety of chemicalencoding data.

A hyperspectral imaging system integrating DLP technology to illuminatethe area of interest with band pass of light ranging over a spectralrange from the visible to the near-infrared that is interfaced with adigital focal plane array for acquisitioning and storing a series ofdigital spectroscopic images (a hyperspectral data cube) that isanalyzed using chemometric methods providing chemical information withinthe area of interest aiding the clinician in detecting, diagnosing andmonitoring disease. Clinicians could also monitor patients with thistechnology for determining the best pharmacological therapy and maximizefavorable patient outcomes. The present invention utilizes DLPtechnology with a focal plane array and chemometric deconvolution forvisualizing the biochemical levels within a human non-invasively forclinical and surgical applications. The present invention can also beilluminated with spectrum of light.

As shown in FIG. 70A, a single column 411 of mirrors may be positionedin an “on” position while the remaining columns are positioned in an“off” position. The resulting spectrum is shown in FIG. 70B. Similarly,as shown in FIG. 71A, a single column 416 of mirrors may be positionedin an “on” position while the remaining columns are positioned in an“off” position. The resulting spectrum is shown in FIG. 71B. It isapparent to one of ordinary skill in the art that the selection ofindividual columns of mirrors can control the output spectrum of thepresent invention, and by varying the selected column, such spectrum canbe shifted within the range of the present invention, i.e., thedifference of the positions of the spectrum in FIGS. 70B and 71B.Furthermore, unlike all other known devices, the present invention canalso provide complex spectra for illuminating a target. For example, asshown in FIG. 72A, various columns of mirrors 411, 412, 413, 414, 416,417 and 418 can be positioned in an “on” position while column 415 ispositioned in an “off” position thereby resulting in a complex spectrum.Moreover, a different number of mirrors within a particular column canbe positioned in an “on” position thereby providing a differentintensity of a particular wavelength of light. Thus, the complexspectrum created by the mirror positioned depicted in FIG. 72A is shownin FIG. 72B.

The DLP® hyperspectral imaging and chemometric deconvolution can beapplied to all products that use light, for example, the fields ofclinical endoscopy, clinical chemistry, microscopy, surgical microscopy,drug discovery, microarray scanners and microplate readers.

It should be appreciated that software included in some embodiments ofthe present invention can perform a variety of functions. For example,the software can provide a user interface through which a user cancontrol all aspects of illumination, measurement and analysis. Thus, thesoftware can provide full control of all hardware devices. Moreover, thesoftware can perform mathematical functions based on internallyprogrammed or embedded functions or can call math functions from outsidesoftware, e.g., MatLab.

It should also be appreciated that the present invention can be used tocompare hyperspectral images of a target of interest, e.g., a patient'skidney, with images of know targets to evaluate the condition of thetarget of interest. For example, a hyperspectral image of a kidney canbe obtained with the present invention and then compared against alibrary of kidney images including but limited to normal kidneys,ischemic kidneys, cancer containing kidneys, etc. Such comparison allowsa medical professional to evaluate the condition/state of a target bycomparison to an image library of similar targets in various conditions.Thus, a protocol for repairing/healing the target can be easilydetermined based on the closest match from the image library.

It should be appreciated that the range of usable wavelengths for thepresent invention is limited when using currently available digitalmicromirror devices, i.e., the physical size of each respective mirrorelement and the materials and coatings used on the front surface windowswhich protect the DMD limit the range of wavelengths. For example, thesize of the mirrors limits what wavelengths will properly reflect fromtheir surfaces, while the window materials and coatings limit whatwavelengths of light can pass therethrough. The present invention, asclaimed, is not limited by such DMD characteristics and therefore as newDMDs become available, they will be directly applicable to the presentinvention.

It is contemplated that any embodiment discussed in this specificationcan be implemented with respect to any method, kit, reagent, orcomposition of the invention, and vice versa. Furthermore, compositionsof the invention can be used to achieve methods of the invention.

It will be understood that particular embodiments described herein areshown by way of illustration and not as limitations of the invention.The principal features of this invention can be employed in variousembodiments without departing from the scope of the invention. Thoseskilled in the art will recognize, or be able to ascertain using no morethan routine experimentation, numerous equivalents to the specificprocedures described herein. Such equivalents are considered to bewithin the scope of this invention and are covered by the claims.

All publications and patent applications mentioned in the specificationare indicative of the level of skill of those skilled in the art towhich this invention pertains. All publications and patent applicationsare herein incorporated by reference to the same extent as if eachindividual publication or patent application was specifically andindividually indicated to be incorporated by reference.

The use of the word “a” or “an” when used in conjunction with the term“comprising” in the claims and/or the specification may mean “one,” butit is also consistent with the meaning of “one or more,” “at least one,”and “one or more than one.” The use of the term “or” in the claims isused to mean “and/or” unless explicitly indicated to refer toalternatives only or the alternatives are mutually exclusive, althoughthe disclosure supports a definition that refers to only alternativesand “and/or.” Throughout this application, the term “about” is used toindicate that a value includes the inherent variation of error for thedevice, the method being employed to determine the value, or thevariation that exists among the study subjects.

As used in this specification and claim(s), the words “comprising” (andany form of comprising, such as “comprise” and “comprises”), “having”(and any form of having, such as “have” and “has”), “including” (and anyform of including, such as “includes” and “include”) or “containing”(and any form of containing, such as “contains” and “contain”) areinclusive or open-ended and do not exclude additional, unrecitedelements or method steps.

The term “or combinations thereof” as used herein refers to allpermutations and combinations of the listed items preceding the term.For example, “A, B, C, or combinations thereof” is intended to includeat least one of: A, B, C, AB, AC, BC, or ABC, and if order is importantin a particular context, also BA, CA, CB, CBA, BCA, ACB, BAC, or CAB.Continuing with this example, expressly included are combinations thatcontain repeats of one or more item or term, such as BB, AAA, MB, BBC,AAABCCCC, CBBAAA, CABABB, and so forth. The skilled artisan willunderstand that typically there is no limit on the number of items orterms in any combination, unless otherwise apparent from the context.

All of the compositions and/or methods disclosed and claimed herein canbe made and executed without undue experimentation in light of thepresent disclosure. While the compositions and methods of this inventionhave been described in terms of preferred embodiments, it will beapparent to those of skill in the art that variations may be applied tothe compositions and/or methods and in the steps or in the sequence ofsteps of the method described herein without departing from the concept,spirit and scope of the invention. All such similar substitutes andmodifications apparent to those skilled in the art are deemed to bewithin the spirit, scope and concept of the invention as defined by theappended claims.

Thus, it is seen that the objects of the present invention areefficiently obtained, although modifications and changes to theinvention should be readily apparent to those having ordinary skill inthe art, which modifications are intended to be within the spirit andscope of the invention as claimed. It also is understood that theforegoing description is illustrative of the present invention andshould not be considered as limiting. Therefore, other embodiments ofthe present invention are possible without departing from the spirit andscope of the present invention.

What is claimed is:
 1. A hyperspectral imaging system having an opticalpath, said system comprising: an illumination source configured tooutput a light beam, said light beam illuminating a target; a dispersingelement arranged in said optical path and configured to separate saidlight beam into a plurality of wavelengths; a spatial light modulatorconfigured to tune said plurality of wavelengths into a spectrum; anoptical device comprising a detector and configured to collect saidspectrum reflected from said target and arranged in said optical path;and, a processor operatively connected to and configured to control atleast one of: said illumination source; said dispersing element; saidspatial light modulator; said optical device; and, said detector, saidprocessor further configured to output a hyperspectral image of saidtarget, wherein said dispersing element is arranged between saidillumination source and said spatial light modulator, said spatial lightmodulator is arranged to transmit said spectrum to said target and saidoptical device is arranged in said optical path after said target,wherein said detector comprises a plurality of pixels and said processorfurther comprises a deconvolution algorithm configured to normalize saidspectrum at each of said plurality of pixels prior to processing saidhyperspectral image.
 2. The hyperspectral imaging system according toclaim 1 wherein said illumination source is an optically tunableradiation source is selected from the group consisting of: asupercontinuum laser and a light emitting diode.
 3. The hyperspectralimaging system according to claim 1 wherein said illumination source isconfigured to produce said light beam comprising said plurality ofwavelengths ranging from 250 nm to 2,500 nm.
 4. The hyperspectralimaging system according to claim 1, wherein said target is afluorescent target.
 5. The hyperspectral imaging system according toclaim 4, wherein said fluorescent target comprises a naturalfluorescence of a tissue or a fluid.
 6. The hyperspectral imaging systemaccording to claim 4, wherein said fluorescent target comprises anatural fluorescence, a fluorescent dye, a fluorescence resonance energydonor, a fluorescence resonance energy acceptor, a fluorescence quencheror combinations thereof.
 7. The hyperspectral imaging system accordingto claim 4, wherein said fluorescent target comprises a fluorescencemolecule selected from the group consisting of: indocyanine green;5-carboxyfluorescein (5-FAM); 6-carboxyfluorescein (6-FAM);fluorescein-5-isothiocyanate (FITC);2′7′-dimethoxy-4′5′-dichloro-6-carboxyfluorescein (JOE); rhodamine andrhodamine derivatives such as N,N,N′,N′-tetramethyl-6-carboxyrhodamine(TAMRA); 6-carboxyrhodamine (R6G); tetramethyl-indocarbocyanine (Cy3);tetramethyl-benzindo carbocyanine (Cy3.5);tetramethyl-indodicarbocyanine (Cy5); tetramethyl-indotricarbocyanine(Cy7); 6-carboxy-X-rhodamine (ROX); hexachloro fluorescein (HEX);tetrachloro fluorescein TET; R-Phycoerythrin;4-(4′-dimethylaminophenylazo) benzoic acid (DABCYL);5-(2′-aminoethyl)aminonaphthalene-1-sulfonic acid (EDANS); and,combinations thereof.
 8. The hyperspectral imaging system according toclaim 1, wherein said target comprises a structure selected from thegroup consisting of: a tissue in a bloodstream; a tissue in a lymphaticsystem; a tissue in a bodily secretion; a tissue in a biopsy; a skintissue; a tissue of a blood vessel; a retinal tissue; a tissue imaged bylaparoscopy; a tissue imaged by endoscopy; a central nervous systemtissue; a bodily fluid; an organ; and, combinations thereof.
 9. Thehyperspectral imaging system according to claim 1, wherein saiddispersing element is selected from the group consisting of: a grating;a prism; a tunable filters; an electromechanical optical filter wheel;an acousto-optical tunable filter; a liquid-crystal tunable filter; adigital micromirror device; an electro-optical filter; a holographicfilter; and, combinations thereof.
 10. The hyperspectral imaging systemaccording to claim 1, wherein said spatial light modulator comprises adigital micromirror device.
 11. The hyperspectral imaging systemaccording to claim 1, wherein said spectrum comprises a narrow bandwidthof wavelengths of said light beam.
 12. The hyperspectral imaging systemaccording to claim 1, wherein said spectrum comprises a complexcomposition of wavelengths of said light beam and respective portions ofsaid complex composition each comprise an intensity.
 13. Thehyperspectral imaging system according to claim 1, wherein said opticaldevice comprises a camera.
 14. The hyperspectral imaging systemaccording to claim 1, wherein said detector is selected from the groupconsisting of: a spectrometer; a two-dimensional array detector; amulti-array detector; an on-chip amplification CCD camera; aback-illuminated CCD camera; a liquid nitrogen cooled focal plane arraycamera; a photomultiplier tube; a focal plane array; a CMOS array and,combination thereof.
 15. The hyperspectral imaging system according toclaim 1, wherein said processor is configured to perform at least oneof: tune said spatial light modulator; trigger said detector forcollection of a series of spectroscopic images formatted as ahyperspectral image cube; and, process said hyperspectral image cube forvisualization.
 16. The hyperspectral imaging system according to claim1, wherein said processor comprises an image data acquisition softwareconfigured to perform at least one of: tune said spatial lightmodulator; trigger said detector for collection of a series ofspectroscopic images formatted as a hyperspectral image cube; and,process said hyperspectral image cube for visualization, said processorfurther comprises a converter to digitize said hyperspectral image cube,wherein said processor is connect to a display.
 17. The hyperspectralimaging system according to claim 1, wherein said processor furthercomprises: a computer comprising an image data processing softwareconfigured to process digital signals; a digital signal processorcomprising an algorithm configured for chemometric analysis; and, analgorithm library configured for visualizing chemistry on and withinsaid target for detection, monitoring and diagnosis.
 18. Thehyperspectral imaging system according to claim 1, wherein saidprocessor is arranged to communicate with a database comprising alibrary of spectra of fluorescent targets, a library of spectra ofabsorbent targets, a library of spectra of reflective targets, a libraryof spectra of organs, a library of spectra of tissues, a library ofspectra of physiological substances or combinations thereof.
 19. Thehyperspectral imaging system according to claim 1, wherein saidprocessor is configured to process said hyperspectral image at a videorate.
 20. The hyperspectral imaging system according to claim 1, whereinsaid processor is configured to form said hyperspectral image in lessthan 1.0 second.
 21. The hyperspectral imaging system according to claim20, wherein said processor is configured to form said hyperspectralimage in approximately 0.33 second.
 22. The hyperspectral imaging systemaccording to claim 1 further comprising illumination optics selectedfrom the group consisting of: a lens; a microscope; a dissectionmicroscope; a surgical microscope; a slit lamp; an endoscope; alaparoscope; a colonoscope; funduscope; broncialscope and, combinationsthereof.
 23. The hyperspectral imaging system according to claim 1,wherein said hyperspectral imaging system is configured for non-invasivereal-time in vivo or in vitro analysis of said target.
 24. Thehyperspectral imaging system according to claim 1 further comprisingcollimating optics configured to transmit said light beam from saiddispersing element to said spatial light modulator as a collimated lightbeam.
 25. The hyperspectral imaging system according to claim 1 furthercomprising beam shaping optics configured to transmit said light beamfrom said spatial light modulator to said target so that said light beamsubstantially illuminations all of said target.
 26. The hyperspectralimaging system according to claim 1, wherein said hyperspectral imagecomprises chemically encoded data.
 27. The hyperspectral imaging systemaccording to claim 26, wherein said chemically encoded data isdetermined by comparing said spectrum reflected from said target to aknown reflected spectrum, by comparing said spectrum reflected from saidtarget to a previously obtained spectrum reflected from said target, orcombinations thereof.
 28. The hyperspectral imaging system according toclaim 1, wherein said target comprises a biomarker configured to absorb,reflect or fluoresce in response to said spectrum.
 29. The hyperspectralimaging system according to claim 28, wherein said biomarker comprises anaturally occurring substance within said target, a substance introducedinto said target or combinations thereof.
 30. The hyperspectral imagingsystem according to claim 28, wherein said biomarker comprises water,oxyhemoglobin, deoxyhemoglobin, carboxyhemoglobin, lipid, bilirubin,cholesterol, collagen, glucose, central nervous system fluid proteins,saliva proteins, semen or combinations thereof.